This guide walks you through setting up automated software testing from scratch: choosing a tool that fits your project, installing it, writing your first automated tests, running them reliably, and wiring them into a continuous integration (CI) pipeline so they execute automatically on every code change.
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Hands-On Automated Testing with Playwright
- ✔ Topic: Automated web application testing
- ✔ Framework: Microsoft Playwright
- ✔ Format: Book / hands-on guide
AI for Quality Assurance and Software Testing
- ✔ Topic: AI in software testing and quality assurance
- ✔ Format: Book / practitioner’s guide
- ✔ Audience: Experienced QA professionals
Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation
- ✔ Series: Addison-Wesley Signature Series (Fowler)
- ✔ Format: Book / professional reference
- ✔ Topic: Build, test, and deployment automation
By the end, you will have a working automation setup that runs a suite of tests against your application and reports pass or fail results without manual effort.
This guide is written for developers and QA engineers with basic programming experience who currently test manually or have no tests at all. You should be comfortable running commands in a terminal and reading simple code. Expect 3-6 hours for the initial setup; writing a meaningful test suite is ongoing work that you build up over weeks.
Difficulty: Intermediate | Time: 3-6 hours for initial setup, plus ongoing test writing
What You’ll Need
Tools & Materials:
- A computer with admin access and a terminal (macOS/Linux shell, or Windows with PowerShell or WSL)
- A supported runtime: Node.js 18+ (for web/UI testing) or Java 11+/Python 3.10+ (for API or general testing)
- A package manager: npm, pip, or Maven/Gradle
- A code repository hosted on GitHub, GitLab, or Bitbucket
- A code editor such as VS Code
- The application you want to test, running locally or in a staging environment
Knowledge:
- Basic command-line usage (running commands, changing directories)
- Fundamentals of at least one programming language (variables, functions, conditionals)
- Basic Git operations (commit, push)
- Familiarity with how your application is structured (web app, mobile app, or API)
Decide your testing target before installing anything. Tools differ sharply by layer: Playwright or Cypress for web UI testing, Postman/Newman or REST Assured for API testing, Appium for mobile apps. Installing the wrong category of tool is the most common wasted effort, so confirm what you need in Step 1 before proceeding. This guide uses Playwright for concrete examples because it covers web UI and API testing in one package, but the process transfers directly to other tools.
Hands-On Automated Testing with Playwright
If your goal in 2026 is to write automated tests that actually run in CI without flaking, this is where we would start. This guide centers entirely on Microsoft’s Playwright framework, and that focus pays off: instead of surveying a dozen tools shallowly, it walks through building tests that are fast, reliable, and scalable — the three properties that separate automation that gets used from automation that gets disabled after the tenth false failure.Compared with the AI guide, this pick is narrower but more immediately actionable. The AI book helps you decide how to transform a QA practice; this one has you shipping passing tests in the first chapters. That immediacy is why we rank it first for most buyers. Playwright itself has also become the default choice for modern web testing, which means the skills here transfer directly to most current job postings and teams.The tradeoff is framework lock-in. Playwright is web-first, so teams testing native mobile apps or desktop software will need a different tool — and the book will not help them. It also does not address the pipeline around your tests, which is exactly where Continuous Delivery picks up the baton.
Pros:
- Focused entirely on building fast, reliable, scalable automated tests
- Playwright is a current industry-standard framework for modern web apps
- Hands-on format produces runnable test code rather than abstract theory
- Approachable enough for engineers newer to test automation
Cons:
- Tightly bound to one framework; skills do not transfer directly to mobile or desktop testing
- Does not cover the build-and-deploy pipeline surrounding your tests
- Framework APIs evolve, so specific code examples may need adaptation over time
Best for: QA engineers and developers who need to build reliable end-to-end tests for web applications now
Not ideal for: Teams automating native mobile, desktop, or API-only testing, or anyone needing pipeline-level architecture guidance
Bottom line: The most efficient way for most practitioners to go from manual testing to dependable automated web tests.
“The most efficient way for most practitioners to go from manual testing to dependable automated web tests.”
AI for Quality Assurance and Software Testing
This practitioner’s guide takes on the question most QA teams are asking in 2026: what does testing look like when AI can generate, prioritize, and maintain tests? It is the only pick in our lineup that addresses that question directly, covering AI-powered testing techniques, tools, and the broader transformation of testing workflows.The coverage is genuinely comprehensive — and that is both its strength and its risk. Where Hands-On Automated Testing with Playwright teaches one framework deeply, this resource surveys the AI landscape broadly. That breadth suits experienced QA professionals who already know how to write tests and now need a strategy for modernizing an entire practice. Compared with Continuous Delivery, it is more focused on the testing function itself rather than the surrounding release machinery.Two caveats matter. First, it assumes existing familiarity with software testing fundamentals — newcomers should learn the basics elsewhere before tackling AI-augmented workflows. Second, AI tooling evolves faster than any other area of software, so specific tools described here may be superseded; the durable value lies in the techniques and transformation frameworks rather than the tool tour.
Pros:
- Comprehensive coverage of AI-powered testing techniques and tools
- Practitioner-oriented with real transformation strategies, not just theory
- Addresses the strategic layer the other picks do not cover
- Strong fit for teams modernizing legacy QA workflows
Cons:
- Assumes prior knowledge of software testing fundamentals
- AI tools evolve quickly, so some content risks becoming outdated
- Broader and less hands-on than a framework-specific guide like the Playwright book
Best for: Experienced QA leads and engineers planning an AI-driven modernization of their testing practice
Not ideal for: Beginners who have not yet mastered core testing fundamentals, or teams not yet ready for AI tooling
Bottom line: The strategic companion pick for QA professionals who need to understand how AI reshapes testing before choosing tools.
“The strategic companion pick for QA professionals who need to understand how AI reshapes testing before choosing tools.”
Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation
Sometimes the hardest part of test automation is not writing the tests — it is making them part of a reliable, automated release process. This is the problem Continuous Delivery was written to solve. Part of the Addison-Wesley Signature Series curated by Martin Fowler, it treats test automation as one stage in a full pipeline of build, test, and deployment automation, and it remains the reference most engineering organizations reach for when designing that pipeline.Its authority is unmatched in this lineup. Where the Playwright guide tells you how to write a test and the AI guide tells you how to transform your testing practice, this book tells you how the whole delivery machine fits together — including the deployment automation the other two picks deliberately skip. For team leads and architects, that systems-level view is worth more than any single-framework tutorial.The tradeoffs are real, though. This is the broadest and least immediately hands-on of the three picks: you will not finish a chapter with runnable test code, and its examples are conceptual rather than tool-specific. It also predates the current wave of AI-augmented testing, so buyers wanting modern AI coverage should pair it with AI for Quality Assurance and Software Testing rather than expect it here.
Pros:
- Authoritative, widely regarded reference on deployment and release automation
- Curated as part of the Addison-Wesley Signature Series edited by Martin Fowler
- Covers the full pipeline — build, test, and deployment — not just test authoring
- Systems-level perspective that scales from one team to an entire organization
Cons:
- Conceptual rather than hands-on; no runnable test code or tool walkthroughs
- Test automation is one topic among many, so coverage is less detailed than dedicated guides
- Predates current AI-powered testing practices
Best for: Engineering leads and DevOps-minded teams designing an automated build-test-deploy pipeline around their tests
Not ideal for: Individual contributors who need to write their first automated tests this week
Bottom line: The foundational reference for teams that want automation embedded in a reliable end-to-end release process.
“The foundational reference for teams that want automation embedded in a reliable end-to-end release process.”
As an Amazon Associate we earn from qualifying purchases.
Before You Start
Do not attempt to automate everything at once. Your first goal is a small suite of 5-10 stable tests covering one high-value flow (for example, login, or a core purchase path). Trying to automate an entire manual regression suite on day one almost always produces flaky, abandoned tests.
Run the target application locally and confirm you can reach it manually in a browser or API client before writing any test. If the app itself is broken, you will waste time debugging tests that are actually failing because of the application.
Check that your CI provider has a free tier sufficient for your build minutes (GitHub Actions and GitLab CI both offer generous free allowances for public and small private projects).
Step-by-Step Instructions
Step 1: Choose the tool that matches your testing layer
List the layers of your application you need to test: web UI, API/backend, or mobile. Then match each layer to one tool using this shortlist:
- Web UI: Playwright or Cypress (both JavaScript/TypeScript; Playwright also supports Python, Java, and .NET)
- API: Playwright API testing, Postman with Newman, or REST Assured (Java)
- Mobile: Appium
- Unit/integration (code level): Jest, pytest, or JUnit, paired with your framework
Pick exactly one tool for your first setup. If your app is a typical web application, choose Playwright: it handles UI and API tests, runs in multiple browsers, and includes a code generator that speeds up test creation.
Tip: Check the tool’s official documentation and release activity before committing. A tool with no releases in over a year will cause dependency problems later.
Check: You have written down one tool name and the reason it fits your application type.
Step 2: Install the tool and its dependencies
Open a terminal in your project root and install the tool using its official installer command. For Playwright with Node.js:
npm init -ynpm install -D @playwright/testnpx playwright install
The last command downloads the browser binaries the tests will drive, and can take several minutes. For other tools, follow the install command from their documentation exactly — do not substitute versions from third-party tutorials.
Commit the package manifest (package.json, requirements.txt, or pom.xml) to your repository so teammates and CI get identical versions.
Tip: Install browsers only through the tool’s own installer. Manually installed browsers often fail version checks and produce confusing errors.
Check: Running the tool’s version command (for example, npx playwright --version) prints a version number with no errors.
Step 3: Configure the tool for your environment
Create the tool’s configuration file in your project root. For Playwright, create playwright.config.ts and define: the base URL of your running application, the browser(s) to test (start with Chromium only), a timeout of 30 seconds per test, and a retries setting of 1 for CI only.
Keep the configuration minimal at first. Add browsers, parallelism, and reporters after your first tests pass.
Store environment-specific values such as the base URL in variables or a .env file rather than hard-coding them into tests, so the same tests can run against local and staging environments.
Tip: Never commit credentials or secrets into the config file or repository. Use environment variables for test accounts and API keys.
Check: The tool starts without configuration errors — for Playwright, running npx playwright test reports ‘no tests found’ rather than a config error.
Step 4: Record or write your first test
Create a test file (for example, tests/login.spec.ts) and write one test for a simple, stable flow: loading a page and checking something visible, or calling one API endpoint and checking the response.
Use the tool’s code generator to produce a first draft quickly: npx playwright codegen https://your-app-url opens a browser; click through your flow and the tool writes the test commands for you. Copy the generated code into your test file and clean it up: remove hard-coded waits, add meaningful assertions, and name the test after the user behavior it verifies.
Add two or three more tests for adjacent steps in the same flow, but stop at five. Depth in one flow beats shallow coverage everywhere.
Tip: Prefer assertions that check visible outcomes (text appears, element is enabled) over assertions on internal state or exact timing. These survive UI changes far better.
Check: Each test file contains at least one clear assertion, not just navigation commands.
Step 5: Run the tests locally and make them stable
Run the full suite from the terminal: npx playwright test. Review the output. Every test should pass against your local application.
Then run the suite three times in a row without changing anything. If any test fails intermittently, fix it now — locate the unstable wait or race condition before adding more tests. Replace fixed delays with explicit waits for a condition (an element becoming visible, a network response completing), which the tool’s API provides.
A suite that passes three consecutive runs locally is stable enough to move to CI. A suite that fails randomly will waste far more time later than the fix costs now.
Tip: Flaky tests are the number one reason automation efforts are abandoned. Treat any intermittent local failure as a blocking defect in the test, not as noise.
Check: Three consecutive full-suite runs complete with all tests passing and no random failures.
Step 6: Connect the tests to a CI pipeline
Create a CI workflow file in your repository. For GitHub Actions, create .github/workflows/tests.yml that: checks out the code, installs Node.js and dependencies, runs the tool’s install command for browsers, and executes the test suite.
For Playwright, the workflow needs npx playwright install --with-deps to add system libraries the browsers require. Set the CI environment variable for your base URL to point at a test environment or a build that the workflow starts itself.
Commit and push the file. The CI provider should detect the workflow and run it automatically.
Tip: Give CI a longer timeout than local runs — CI machines are slower, and browser startup can take 10-20 seconds per worker.
Check: Opening the CI provider’s web interface shows a workflow run for your latest commit, with the test job marked green and a test-count summary in the log.
Step 7: Verify the pipeline blocks broken code
Deliberately break something: add a temporary change that makes a test fail (for example, change an expected page title in a test or introduce a bug in the app). Push the commit.
Watch the CI run. The pipeline should turn red, the failing test should be named in the log, and — if you configured branch protection — the merge should be blocked.
Revert the change and confirm the next run returns to green. This proves your automation actually catches problems rather than just running.
Tip: If you want failed tests to block merges, enable branch protection rules in your repository settings now; the pipeline alone does not enforce this.
Check: A broken commit produces a red, failed CI run naming the failing test; reverting restores a green run.
Common Mistakes to Avoid
- Automating too many tests at once before the setup is stable — Cap your first suite at 5-10 tests covering one flow. Expand only after the suite passes three consecutive runs locally and one full CI cycle.
- Using fixed sleep/wait commands throughout tests — Replace every fixed delay with an explicit wait for a condition — element visible, text present, or network response received. Fixed waits are the main source of slow, flaky suites.
- Hard-coding URLs, credentials, and environment values in test files — Load these from environment variables or config files from the first test onward, so the same suite runs against local and CI without edits.
- Choosing a tool based on popularity rather than fit with the application and team’s language — Match the tool to the testing layer (UI, API, mobile) and to a language your team already writes. A powerful tool in an unfamiliar language becomes unmaintained.
Troubleshooting
Problem: Tests pass locally but fail in CI with browser or library errors
Solution: Run npx playwright install --with-deps (or your tool’s equivalent) in the CI workflow so system libraries and browsers are installed on every run. Also confirm the Node/Python/Java version in CI matches your local version.
Problem: A test fails intermittently with no code changes (flaky test)
Solution: Run the single test repeatedly with the tool’s repeat flag (Playwright: --repeat-each=10). Identify the failing assertion, then replace timing assumptions with waits on real conditions. If the app itself is slow under load, increase that test’s timeout rather than adding global delays.
Problem: CI cannot reach your application (connection refused, timeout)
Solution: CI has no access to localhost. Either configure the workflow to build and start your app as a job step, point the base URL at a staging environment reachable from CI, or use a service container for the backend. Set the base URL via environment variable, not in the test code.
Problem: Selector-based tests break every time the UI changes slightly
Solution: Replace brittle selectors like nth-child chains or auto-generated class names with stable identifiers: accessible roles, labels, data-testid attributes, or visible text. Add data-testid attributes to your own components where stable selectors do not exist.
What Success Looks Like
Your setup is complete and working when all of the following are true:
- A test suite of 5 or more automated tests runs from a single terminal command and finishes with all passing.
- The same suite runs automatically in CI on every push, without manual triggering.
- A deliberately broken change produces a red CI run that names the failing test, and reverting restores a green run.
- Three consecutive suite runs, local and CI, produce identical results with no random failures.
- Configuration, URLs, and credentials live in version-controlled config and environment variables — nothing hard-coded in test files.
Next Steps
With the pipeline running, grow coverage deliberately. Add tests for the next high-value user flow each week, prioritizing paths where manual testing takes the most time. Add a test reporter (HTML reports work well for sharing results with non-developers) and screenshot or video capture on failure to speed up debugging. Once UI coverage is solid, add API-level tests for business logic — they run faster and break less often than UI tests. Set up a nightly scheduled run to catch environment drift, and review flaky tests weekly: quarantine or fix them so the red/green signal stays trustworthy. When your suite exceeds a few hundred tests, parallelize CI runs to keep feedback under ten minutes.
Frequently Asked Questions
Do I need to know how to code to use test automation tools?
For a maintainable setup, yes — basic programming in one language is needed, because recorded tests always require cleanup and debugging. Low-code tools can record flows without coding, but recorded scripts become brittle quickly and are best treated as drafts to refine, not finished tests.
Which should I automate first: UI tests or API tests?
Automate the layer where manual testing hurts most. If testers spend hours clicking through checkout flows, start with UI. If the pain is verifying backend behavior across many data combinations, API tests deliver more coverage per hour of effort and are less prone to breaking. Ideally you end up with more API tests than UI tests over time.
How many automated tests do I need before the setup is worthwhile?
Quality matters more than count. A stable suite of 10 tests covering your most repeated manual checks already saves time every release. A suite of 500 flaky tests saves nothing because nobody trusts the results. Grow coverage steadily and keep every test green.
What should I do about tests that fail only sometimes?
Treat flakiness as a defect. Reproduce it by running the test repeatedly, identify whether the test is racing the application (fix with condition-based waits) or the app is genuinely inconsistent (file a bug). Most teams quarantine known-flaky tests in a separate suite so they do not erode trust in the main pipeline.
Can automated testing replace manual testing entirely?
No. Automation verifies known expected behavior efficiently; it cannot discover unexpected problems, judge usability, or explore edge cases the way a person can. Use automation to eliminate repetitive regression checking so manual testing time goes to exploratory and usability work.
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
