What is the difference between SeaLights and SonarQube?
SeaLights and SonarQube are complementary tools.
“SonarQube is an open-source platform developed by SonarSource for continuous inspection of code quality to perform automatic reviews with static analysis of code to detect bugs, code smells on 29 programming languages. SonarQube offers reports on duplicated code, coding standards, unit tests, code coverage, code complexity, comments, bugs, and security recommendations.” Wikipedia, https://en.wikipedia.org/wiki/SonarQube
SeaLights is a testing optimization platform focused on software product quality by helping enterprises optimize their testing development and execution efforts. Using binary code instrumentation SeaLights offers reports on code coverage, modified code coverage and testing gaps for all test types, specifically non-unit tests. These reports allow companies to assess and improve their testing development strategy. SeaLights also provides recommendations of which tests should be executed according to the code that was actually changed, thus speeding up the test execution cycle without compromising on product quality.
While SonarQube is targeted for developers working locally on unit tests, SeaLights enables QA owners and executives to gain visibility and quality improvement across all testing efforts.
How do SeaLights and SonarQube complement each other?
|  | |
Target audience | Developers working locally on unit tests. | QA owners, manual testers, automation engineers, QE teams, product owners, executives - focused on different testing efforts across the entire pipeline. |
Code coverage & modified code coverage measurement | For unit tests. Can be set up for integration tests (on a limited Maven setup) or for manual tests but requires effort to configure. | SeaLights captures coverage from the entire business flow through all the methods and other services, for all test types, specifically non-unit and manual tests. This includes: integration tests, API tests, nightly tests, regression tests, manual tests, e2e tests and more. |
Test gap analysis | Shows gaps of code not covered by unit tests. | Provides a detailed overview of changed / unchanged code that was not tested in all / specific test stages / types. Meaning that this overview is provided across the entire testing cycle, and is not just limited to unit tests. |
Code coverage analysis | Shows coverage data only for the current build (out of the box). | Coverage data can be broken down or aggregated across time, testing stages/types, builds, sprints and teams. This allows you to monitor your code coverage and analyze the efficiency of your testing strategy. |
Quality gates | For Pull Request / unit tests only. | Policy-based across the entire pipeline, when promoting from one environment to the next. |
Coverage trend analytics | No out of the box reports. | Coverage trend reports with automatic insights are available across time, testing stages/types and teams. Assisting you to perform analysis for a single component level all the way up to an executive level overviewing the entire product or organization. |
Test prioritization | No. | Automatically recommends which tests are not impacted by the code changes and can be skipped / executed later in order to save time, resources and fail faster. |
Test optimization | No. | Using integration to multiple testing frameworks, automatically skips unimpacted tests according to a pre-set testing policy. Provides reports on the tests executed and skipped, including compute time saved. |
Out of the box risk notifications | No out of the box notifications. | Policy-based notifications via email or Slack. |
SOC2 Type II Compliance | Please check with the vendor. | SOC2 Type II annual certification for the entire solution. |
Source code required access | Requires full access to the entire code base. | No need to access the code base. Only scans binaries and artifacts. |
Code scan technology | Static code analysis which requires full access to the entire code base. | Binary code analysis. No access to the source code is required, thus also enables to monitor 3rd party libraries. Scan is performed by agents developed in-house to provide comprehensive customer support. |
Capturing test coverage | Relies on 3rd party Open Source tools integrations (JaCoCo, Istanbul, etc.), available mainly for unit tests. Can be set up for integration tests (on a limited Maven setup) or for manual tests but requires effort to configure. | Test coverage is captured by agents developed in-house to provide comprehensive customer Support, and available for any test type. All supported languages use dynamic code instrumentation during run time on the testing machine, except for JavaScript & Go which use static instrumentation and update the artifacts themselves to include the coverage capturing code. |
Coverage level | Coverage is captured on a line and code branch level. | Coverage is captured on a method level. For some technologies, coverage is also captured on a line or code branch level. |
SeaLights supports unit test coverage. Isn’t it the same coverage as SonarQube provides?
No. While SeaLights provides unit test coverage, its scanning and calculation mechanism is different than SonarQube’s:
The SonarQube code coverage relies on third-party tools (such as JaCoCo for Java applications and Visual Studio / Dot Cover / Open Cover for .NET). That means that each component acts differently, has its own edge cases and produces different types of reports. In SeaLights, all the agents that scan the binaries and capture the coverage are developed internally in order to follow similar guidelines and provide consistent reports (where possible) across different components and machines.
Some third-party tools such as JaCoCo calculate coverage just for files that have been loaded into memory, and are filtering parts of the code according to some configuration. SeaLights calculates coverage for the entire code base and allows you to configure which parts of the code should be included/excluded in the calculation.
SonarQube provides code line level coverage, while SeaLights usually provides method level coverage.
Due to the above, you should expect to see differences in the coverage provided by SonarQube vs. the coverage provided by SeaLights for unit tests. Since SonarQube also provides additional code insights, if you are using it today we recommend to keep using it for unit test coverage and use SeaLights for other test stages coverage (as well as additional features such as test optimization).
Interested to learn more? Read the following article by our engineering manager, Nadav Yeheskel: https://www.sealights.io/coverage-collection-sealights-vs-sonar/
Why does SeaLights provide method-level coverage?
Unlike unit test coverage where you are testing only the specific methods and therefore need that higher granularity, SeaLights captures coverage from the entire business flow through all the methods and other services. This means that even though there will be situations where methods are considered covered when not all their branches were covered, all the methods those uncovered lines call will not be covered and will be reflected through them in the modified coverage (and they will appear in the testing gap analysis that SeaLights provides).
We have found from our experience that the gain in accuracy is small whereas the increase in difficulty of investigation is major and therefore recommend to concentrate on the method level. This is because SeaLights concentrates on the modified coverage which is the highest risk area. Looking at line coverage will increase the number of untested modified code to a number that will be hard to work on and that discourages teams from closing their testing gaps and raising their quality. Our customers that only used method coverage successfully increased their quality and reduced production issues by over 90% without the need to work with line coverage.