QA Blog Posts
Red Herring in Test
A Red herring is is when an irrelevant topic is introduced during an argument to divert the participants from the original issue. Sometimes this is used to make false assumptions. This is commonly used in books and drama TV to mislead the viewer of the actual issue.
Red Herring in Test
In testing, a Red Herring is when multiple issues are found around common functionality. The multiple issues are the Red Herring. The core feature is what should be addressed.
Example Situation #1
Roxana is testing a new login page, and notices that the Credit Card field is not doing any server-side validation. In further testing, it is discovered that users can submit an form with an empty name and zip code. Separate tickets are filed for those issues.
The important question in this situation is the form field validation. What are the requirements and why are the "basic" functionality not set up.
Example Situation #2
During weekly testing its discovered that the local server keeps running into issues. The Release Engineering team fixes the issue and QA is able to move forward.
When testing, Its important for QA to look at the big issue and not get caught up with Red Herrings that can be distracted from the main issue.
QA Manager Task List
Things that QA Managers should be doing on a constant basis:
- Planning, prioritization of all the test-related tasks (use proven project management tools such as the Waterfall or Scrum methodology)
- Writing the world's best test strategies
- Reviewing the test plans with Product and lead engineers
- Taking the responsibility of certain designs if people have not the required competencies
- Automation Code reviewing
- Spreading expertise and good usage of tools such as bug-tracking database or data capture for automation
- Having people judgment skills to hire the right people
- Writing performance review
- Evaluating the previous release - what went well, and ways that QA could have done things better.
What makes a good QA Manager
- Being a good QA engineer - Be an excellent tester and delegate testing.
- Effective communication - Let the team know what going on and make sure there are no bad surprises as code freeze approaches.
- Having and spreading the "customer-focus" vision - Always make sure that that the customer is happy.
- Developing people - Encouraging people to learn more about the software and industry
- Bringing out the creativity in others - Asking open-ended questions in meetings to generate a good team discussion.
- Motivating people - Everyplace: Meetings, Hallways and Slack channels.
- Team Building - Make sure that everyone on the team works together. Always look for ways to involve several team members in testing functionality.
- Enabling Changes - Constantly looking to improve the testing strategy and techniques. If you're using the same test case repository for the past five years - you're not enabling change.
- Decision Making - Make key decisions about if a feature is ready for release. Sometimes the decision is easy, many times it may get some push back from Engineering, Product or Marketing.
CASL stands for Canada Anti-Spam Law. The goal of the law is to protect Canadians from unwanted emails.
Under the law, Websites that serve Canada Citizens are required to have consent before sending a commercial electronic message. Failure to get consent means the company will have to pay a $48,000 fine and develop a compliance program that includes staff training and policy changes.
Configure the Browser as a Canadian User
One way to trick a browser's actual location is to change the browser's default language. Many sites use this to determine if the CASL message should be displayed.
Configure Google Chrome
Simple steps to change Google Chrome settings. You may want to set up a different profile (Canadian User) for future testing purposes.
- Go to Preferences
- Search for Language
- Click on Language
- Click Add languages, search for Canada and select English (Canada)
- Click Add
- Click on the 3-dots to the right of English (Canada) and select "Move to top"
I would recommend restarting the browser so the setting takes effect.
Simple steps to change the default browser settings to be Canadian English.
- Go to Preferences
- Search for Language
- Select the pull down "Search for more languages"
- Click on the "Select a language to add..." and look for English (Canada)
- Move "English (Canada)" to the top, then click "OK"
I would recommend restarting the browser so the setting takes effect.
Couple of Console Tricks
Here are a couple of little known Google Console tools that might assist with testing websites. These are in the Rendering section of the Google DevTools section.
What's weird is that Google doesn't provide much information about these tools. In 2016, Rendering was hidden from general Developer view.
Highlight Ad Frames
This should be enabled so that you can see what areas of the page that have the potential to being blocked by AdWare tools. While it won't catch 100% of the instance that a piece of content may get blocked, it's good to have as a "hey check this out."
Emulate CSS Media
With this enabled, you can save some steps and see what the page looks like when printed.
In addition, this makes it so much easier to debug a problematic page. For example, if a test position, you can list the exact issue in the bug report - saving time for everyone.
Letter to the QA Manager
A letter to the QA Manager. This seems appropriate on this Labor Day:
Dear QA Manager,
I love my job and I think you are a good quality assurance manager, but there are some things I would like to hear from you and some things I would like you to do.
I want you to give me praise. I want to know when I have done a good job. Don't let the only time you have a serious conversation with me be when I have done something wrong. Let me know when I have helped make the release a success.
Tell me what my job is, how to do it and what QA rules I need to follow. I want to clearly understand what my job is, how you want me to do it and what policies I need to follow. Be specific and give me clear instructions.
I want to be excited by my work. I don't want a dull, boring job where I never know what is going on. Get me excited about what my role is, what our project is, about our department and our company. Show me how I fit into the plan.
Challenge me. I don't want to keep doing the same job each day. Give me something new to do. Challenge my skills and my mind. I can contribute more than you think.
Give me opportunities to improve myself. I am a good worker, but I want to be a better one. Train me in something new, send me to a class, let me learn a new skill from a coworker. Help me develop my career. Help me to be better able to help you.
Delegate to me. When you need something done, let me do it. Let me prove that I am capable of doing more. Trust me to get the job done right and on time. It will build my confidence and respect that you can trust me with a task.
Deal with my coworker. You explained the policies I must follow, but you need to back it up by enforcing those policies. Be fair and treat everyone equally. It kills my morale when a coworker is not disciplined for consistently breaking the rules.
Inspire me. Show me you are passionate about your work, about our team, about our project. Share your vision for our department and our team. Tell me what our mission statement is and why it is important.
Ask for my opinions and ideas. I have worked here for a long time and I know what works and what doesn't. Ask me how things can be improved and what we can do to make our team and our department more productive and successful.
Improve my work environment. My desk and chair are good. My computer is old and slow. I need a new one with a monitor that does not hurt my eyes. You need to remodel our break area. I will be happier and be more productive in an improved work environment.
Fake Referer for Testing
This is a sample QA task that I had many years ago:
To perform this task, I used the Unix curl tool to get the job done.
This "hack" is useful when you want to fake a referer from a particular website, you can "dump" the results as an HTML file to verify the results was a success. It also helps if you have access to the apache logs to verify the referer was pass through.
CURL Config File
curl --config test.txt
In the config file:
# --- Example file --- url = "http://www.cryan.com/Boston/" output = "cia.html" referer = "http://www.cia.gov" user-agent = "superagent/1.0" # and fetch another URL too url = http://www.cryan.com/Boston/" output = "yahoo.html" referer = "http://www.yahoo.com" # and fetch another URL too url = "http://www.cryan.com/Boston/" output = "bing.html" referer = "http://www.bing.com"
This is what you see when you run the command:
curl --config sample.txt % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 10813 0 10813 0 0 35677 0 --:--:-- --:--:-- --:--:-- 35686 100 315 100 315 0 0 3924 0 --:--:-- --:--:-- --:--:-- 3924 100 311 100 311 0 0 3905 0 --:--:-- --:--:-- --:--:-- 3905
You can learn more cool tricks with CURL on the "man" page: curl - How To Use
Testing software may involve testing certain locations. Chrome uses various means to track your location and pass that information to websites. Some of the data is based on Wifi and browser settings.
Location Guard allows you to fake your location. This makes it easy to test the functionality of the site against various locales around the world. For example, Canada has rules about requesting email on websites.
Location Guard is a browser extension that allows to protect your location while using location-aware websites, by adding controlled noise to it.
Latest updates to Chrome might have made this extension non-working. According to many reviews, the extensions are not working 100% of the time.
I'll test it against Google Maps and other sites to see if it still not working.
QA is not Medical Science
In a job interview that I had many years ago at a popular software company, my future boss told me this:
"If bugs happen that's ok, this isn't Medical Science. No one will die because a big made it into Production. "
I was surprised by her comment and took it as the wrong attitude towards excellence in testing. Isn't your goal to make sure bugs don't make it into production?
QA Realistic Goals
However, over the years I have learned she was right. In a fast pace environment, QA can't always deliver 100% bug-free releases. When bugs do make it to production - it's important to avoid the blame game.
QA should focus on how to respond to fixing bugs in production. Once the issue is fixed, QA then needs to find out how it missed the bug - so that it doesn't happen again.
Bugs will happen, but it how you respond to getting the bug fix determines how good of a QA you are.
In testing, a False positive happens when a particular test passed when in fact that it should have failed. This usually happens when tests are not clearly written to catch errors.
Having a lot of failures missed by automation will certainly question the overall value automation brings to the team.
While most people think of False Positive in software automation or medical tests. Here’s a unique perspective to False Positive.
Haunted Mansion Ride Example
At Disneyland Haunted Mansion, Disney has a lot of sensors all over the ride. They are set up to make sure foreign objects are not thrown from the doom buggy. When an object is detected the ride stops to alert security.
Every once in a while, someone manages to throw something into the graveyard without the sensors going off.
The false positive in this situation is that the motion sensors indicate everything is fine and the ride should continue.
The situation is usually fixed within 24-hours but the person responsible got away with it.
Audit Tests that pass
If tests are consistently passing you may want to audit them. Perhaps the test is too specific and may need some additional validation statements to help make the test more efficient.
On vacation this week, so I thought I would keep things simple by posting some recent QA memes.
I will let QA know before pushing to Production
Next Week we’ll return with some more quality QA postings.
A Test Strategy document conversion the overall test approach and goals to a project. A test plan covers how specific features/functionality will be tested.
A test strategy is useful reference project document to highlight the testing coverage of a major new feature. A test strategy should be done with every major project. Small projects and bug fixes are not something that needs a test strategy document.
Test Strategy Topics
Here are the main categories of a Test Strategy Document:
- Goal - What is the goal of the particular test strategy
- Sign-Off - Have some sign-off by the QA Lead, Dev Lead, Business Owner and Project Architect
- Scope - Identify the product features and risk areas. Include important milestones.
- Test schedule - Know when the project is going live and any rollout plan.
- Release Testing - Cover testing that will be performed at each release deployment. Good to keep a history for tracking purposes.
- Schedule Risks - What are areas of the project that could impact the release of the feature. Example might be third party support, special configurations, account settings.
- QA Resources - Who should people contact if there are issues?
- Timeline - Similar to the Test Schedule, but more focus on the project, what's QA involvement in the release of the feature.
- Jira Tickets - List of the story tickets for this project.
- Project Library - Links to the important documents to this project.
- Competition - List of similar features that are done in other companies. This is useful to understand how consumers may view this product.
Three Tips on Test Strategy Document
This is a live document and the QA Lead should be updating the document - at least once per week. Timelines and releases are areas that get the most updates.
Once the Test Strategy document structure is in place, the document should be shared with the project stakeholders Once it's been signed-off then the document should be shared with other QA Members.
Once the project is completed - usually when the project maintenance has been completed, the document should conclude with a "Transition Plan." This is when the test cases become part of the general release testing. Usually the QA Lead would go through the test cases to tag items that should be automated for regular regression.
Test Plan Guidlines
A test plan provides a detailed outline of testing for a particular project in a particular release.
Having a carefully well written test plan can help ensure a feature gets well tested at every release cycle.
There are several key parts of the Test Plan:
This is where the QA Lead identifies various conditions which makes it possible to test the functionality.
This is where you list the conditions that can't be known a for sure ahead of time, which you have to make assumptions. These need to be stated so that QA can evaluate whether they are reasonable.
- Database contains the correct data.
- SQL Integration Service validates the data being inserted into the database.
- Users can only setup one category for the Employee Update page.
Dependencies that this test plan has on external conditions.
- Nightly backup of external HTML XML Feed data
- Intranet User Authentication
Risks & Issues
Conditions that add risk to the quality of the functionality or that may impact the accuracy of the test plan.
Example Risk Points
- SQL Server Integration Service is unable to read the XML Feed
- Poorly written specs
- The business owner is not clearly defined for this project.
- The current database schema is not available.
- QA isn't aware of the technology being use to display the data on the page.
In several paragraphs explain in detail the feature that is being tested. Links to external spec documents and technical documents should be referenced here.
A list of project/developers should be listed here. These are the people that should get tickets of any bugs found when testing this product.
Testing Outline and Summaries
This section should list all the test cases title summary. It would be good to sort the test cases by some sort of topic.
Example Test Case Summaries
- Validate that the XML passes the XML validator test
- Validate that the intranet page is loading after the nightly XML update
- Validate that the intranet page loads if there are no new hires
- Validate that the intranet page loads if there are no upcoming anniversary dates
- Validate that if a user hasn’t set a configuration, the page still loads
According to the National Institute of Health:
A false negative is a test result that indicates a person does not have a disease or condition when the person actually does have it.
In Quality Assurance Automation testing, the National Institute of Health example could be rewritten as:
False Negative happens when a test is run and returns a success when the actual test should have failed.
Why Good Test Produce False Negatives
This can happen because of various reasons:
Hard Coded Information
Sometimes an automation test case may be written to use a specific datasets and/or urls. This type of test case doesn't take into account of actual path that users may use. Taking shortcuts may seem like a good idea, but are risky since they don't take into account the true paths users may take.
Real Life Example
If someone changes a button link url reference, and automation just goes to the URL, without clicking on the button, it will miss that change. The test will pass, but it should have failed.
Not Checking for Errors
In some instances, an error might be thrown to the user - or the browser console during an automation run. Since this isn't directly impacting the flow of the test case, it's considered a "pass."
Real Life Example
Not a Challenging Test Case
I have seen some test cases take the soft route of testing. The test case doesn't check for data validation. Some test cases assume too much and don't intentionally thrown an error - such as entering too much data in a text field.
Real Life Example
Ways to Avoid False Negatives
Four ideas to help reduce the chances of test cases returning False Negatives:
- As a clean up step in automation, check the console logs for errors.
- If using Ghost Inspector, watch the video in some of the critical path testing to see if there's anything out of the ordinary.
- Code review automation test cases.
- In one of my previous jobs, if an Automation test passes for 3-consecutive releases, it's a candidate for an audit. The audit should be done by someone who didn't write the initial case. The audit should think about ways the test could be more productive.
Automation Encountered a New Feature
Ideally the Automation team should get a "heads up" on a new feature being shipped in the current release and areas of impact. Every once in a while, the automation team will be left off the code review or show-me meetings.
Poorly Written Test
The automation steps didn't take into accounts of other situations that may happen when the test are run.
Something happen that cause the server to behave weird - such as a slow network or disk space full. These types of issues are unavoidable.
Best Cheap Beer Social Event
All work and No Play makes QA Testing job boring. One of the popular social events that one of the companies I worked at had was something called the "Best Check Beer Social Event."
The goal of this popular event is to do a taste test to find which of the cheapest beers taste the best. Participants would buy their own been and someone would pour them into plastic cups so that people can do a blind taste.
This is a fun social event to get people to think about what makes beer taste good - and to try to figure out what beer they are tasting.
Here’s what to do:
- Choose your favorite selection from [Beer Advocate's Bottom 100 Beers](https://www.beeradvocate.com/lists/bottom/) (or something else… doesn’t really matter… just has to be cheap)
- Let the organizer know what you plan to bring so there's no overlap.
- Bring in a 6-pack of your beer on taste day (if it only comes in 4-pack, then get two)
- We’ll have a blind taste test and give each beer a 1-10 rating in the following categories:
Then we’ll get someone in product to make some charts and graphs with the results.
Samples Beers Tried in the Past
Here's a list of some of the beers people have brought in the past. I won't give away which beer has been the winner, but I will say that Keystone Light voted pretty well in the blind taste test.
- Miller 64
- High Life
- Miller Light
- Miller Fortune
- The Beast
- Steel Reserve
- Natty Light
- Keystone Light
Did you know that you can create dynamic bookmarks using Bookmarklets?
Some "Real World" Examples:
Atlassian Jira - Show me all the new issues created in the past 7 days. (This could easily be set up as a Dashboard widget - but that's a post for another day.)
Atlassian Jira - Show me all issues that have this week tag, where the tag format tag might look like: deployment-2019-01-07
Google Photos - Show me all photos taken on this day of the year or you can get fancy and say "Show me all photos taken 90-days ago."
Change the URL Server - you can check to see what day it is and change the URL to point to production on release day.
Here's a sample code