testing/android/docs/junit4.md

# JUnit 4 Migration

As of Android 24 (N), JUnit3 style javatests have been deprecated for the new
JUnit4-based [Android Testing Support Library][1].
We are in the process of changing all instrumentation tests in chromium to
JUnit4 style. This doc explains the differences between JUnit3 and JUnit4
instrumentation tests and how to write or convert them.

[TOC]

## Differences between JUnit3 and JUnit4 instrumentation tests

|              | JUnit3                                              | JUnit4                                   |
|--------------|-----------------------------------------------------|------------------------------------------|
| Inheritance  | Tests extends TestCase or child classes             | No inheritance.                          |
| Test methods | methods named with test prefix                      | methods annotated with @Test             |
| Set up       | setUp() method                                      | public method annotated with @Before     |
| Tear down    | tearDown() method                                   | public method annotated with @After      |
| Test runner  | declared within test apk AndroidManifest.xml        | Must specify `chromium-junit4:"true"`    |
| Class runner | N/A                                                 | @RunWith(XClassRunner.class)             |
| Assertion    | Extends from junit.framework.Assert, inherited APIs | Use static methods from org.junit.Assert |

> Please note that during the migration, we support running JUnit3 and JUnit4
> tests in the same apk. This requires two tags, one each for JUnit3 and JUnit4.
> The tag for the JUnit4 runner must specify `chromium-junit4:"true"`
> ([Example][2])

-   **Other JUnit4 features**:
    - Tests can be annotated to expect an exception, e.g.
      `@Test(expected=MyException.class)`. Tests annotated this way will
      fail if they do not throw the given exception.
   - **Test suite set up**: public static method annotated with `@BeforeClass`
   - **Test suite tear down**: public static method annotated with
                                      `@AfterClass`
-  **Replacement for JUnit3 test base classes**
    - [`TestRule`][3]:
        - TestRule is a class to **outsource your test setUp, tearDown, and
          utility methods**. Since there are no more interitance and TestBase classes,
          one should use TestRule for any API calls provided by its test base classes
          previously.
        - One test can declare multiple TestRules and the class runner will run all of
          them. If the order of the TestRule matters to you, use
          [`RuleChain`][8]

    - [`ActivityTestRule`][4]
        - `ActivityTestRule` is a special `TestRule` provided by Android Testing
          Support Library that allows tests to launch an Activity.
          ([Documentation][4])

## Example Code of JUnit3 test and JUnit4 test

JUnit3:

```java
public class MyTestClass extends MyActivityInstrumentationTestCase2<TestActivity> {
    @Override
    protected void setUp(){
        super.setUp();
        setActivityIntent(new Intent());
        getActivity();
    }

    @Override
    protected void tearDown() {
        specialActionFromSuper();
        super.tearDown();
    }

    public void testA() {
        assertEquals(1, 1);
    }
}
```

JUnit4:

```java
@RunWith(BaseJUnit4ClassRunner.class);
public class TestClass {
    @Rule public ActivityTestRule<TestActivity> mRule = new ActivityTestRule<>(TestActivity.class);

    @Before
    public void setUp() { //Must be public
        mRule.launchActivity(new Intent());
    }

    @After
    public void tearDown() { //Must be public
        mRule.specialActionFromActivityTestRule();
    }

    @Test
    public void testA() {
        Assert.assertEquals(1, 1);
    }
}
```

## Migration process

1.  Add required libraries to your target dependencies in BUILD.gn
    -   JUnit 4 library: `//third_party/junit`
    -   Android Testing Support Rules:
        -   `//third_party/android_support_test_runner:runner_java`
            (for `AndroidJUnitRunner`, etc)
        -   `//third_party/android_support_test_runner:rules_java`
            (for `ActivityTestRule`, etc)
1.  Add class runner to your test apk manifest.
    ([example][2])
    -   Keep in mind you can have multiple instrumentations in your manifest.
        Our test runner will run JUnit4 tests with JUnit4 runner and JUnit3
        tests with non-JUnit4 runner.
1.  Refactor TestBase class to a TestRule class.
    ([example CL](https://codereview.chromium.org/2632043002))
    -   +yolandyan will do this part, however, if you did refactoring yourself,
        please add him as a reviewer for your CL and enjoy his eternal appreciation!
1.  Use [auto migrate script][5] to or manually convert all JUnit3 tests to
    JUnit4 style in a your javatest directory
    -   we understand it's tedious to just manually write all the annotations,
        change modifiers, etc to convert all the javatest, so we created an auto
        change script that helps you to convert all the javatests in a certain
        directory. Please check its [README page][5]
        on instructions.


## Customized TestRule example

TestRule:

```java
public class MyRule implements TestRule {
    // 1: Add utility methods...

    @Override
    public Statement apply(final Statement base, Description desc) {
        return new Statement() {
            @Override
            public void evaluate() {
                // 2: Code here runs before @Before method
                base.evaluate()
                // 3: Code here runs after @After method
            }
        }
    }
}
```

## Command Line Flags

In our Junit3 tests command line flags (set by the CommandLineFlag annotations) were inherited from the
test base classes. As an example, ChromeActivityTestBase is annotated with:

```java
@CommandLineFlags.Add({ChromeSwitches.DISABLE_FIRST_RUN_EXPERIENCE, ...
```

and as a result any test in a class derived from ChromeActivityTestBase will disable the first run experience.

The Junit4 tests classes are not however, derived from test base classes; instead their behavior is defined by
test rules. To support this our Junit4 test runner will examine the command line flag annotations on all rules
referenced with @Rule annotations in the test class. In addition, where one rule is derived from another, the
command line flags propogate through the hierarchy of rules. See, for example, [BottomSheetTestRule][11]

Note:- This has only recently been implemented, so is not yet used in all tests. See [this bug][12]

The CommandLineFlags annonations are more fully documented in the [CommandLineFlags class][13]

## Common Errors

1.  Instrumentation tests that rely on test thread to have message handler
    will not work. For example error message:

        java.lang.RuntimeException: Can't create handler inside thread that has not called Looper.prepare()

    or

        java.lang.IllegalStateException: The current thread must have a looper!

    Please utilize `ActivityTestRule.runOnUiThread(Runnable r)` to refactor
    these tests. For more, check this [GitHub issue][6]

1.  Use `@UiThreadTest` with caution!!
    -   Currently, **@UiThreadTest is only effective when UiThreadTestRule or
        ActivityTestRule is declared** in the test class.
    -   Please use **`android.support.test.annotation.UiThreadTest`, NOT
        `android.test.UiThreadTest`**.
    -   When using @UiThreadTest, **it would cause `setUp` and `tearDown` to
        run in Ui Thread** as well. Avoid that by calling [`runOnUiThread`][9]
        or [`runOnMainSync`][10] with a Runnable.
    
    ```java
    // Wrong test
    public class Test {
        @Rule
        public ActivityTestRule<MyActivity> mRule = new ActivityTestRule<>(
                MyActivity.class>

        @Before
        public void setUp() {
            // Cause failure because this also runs on Ui Thread, while it
            // is intended for Instrumentation worker thread
            mRule.launchActivity() 
        }

        @UiThreadTest
        public void test() {
            actionThatNeedsUiThread();
        }
    }
    ```

    The correct thing to do is

    ```java
    // Correct test
    public class Test {
        @Rule
        public ActivityTestRule<MyActivity> mRule = new ActivityTestRule<>(
                MyActivity.class>

        @Before
        public void setUp() {
            mRule.launchActivity()
        }

        public void test() {
            mRule.runOnUiThread(new Runnable() {
                @Override
                public void run() {
                    actionThatNeedsUiThread();
                }
            });
        }
    }
    ```

1.  `assertEquals(float a, float b)` and `assertEquals(double a, double b)` are
    deprecated in JUnit4's Assert class. **Despite only generating a warning at
    build time, they fail at runtime.** Please use
    `Assert.assertEquals(float a, float b, float delta)`

1.  Errorprone expects all public methods starting with `test...` to be
    annotated with `@Test`. Failure to meet that expectation will cause
    errorprone to fail with something like this: 

        [JUnit4TestNotRun] Test method will not be run; please add @Test annotation

    In particular, you may see this when attempting to disable tests. In that
    case, the test should be annotated with both @DisabledTest and @Test.

## Common questions

-   Q: Are `@Test` and `@LargeTest/@MediumTest/@SmallTest` annotation both
    necessary?
    -   A: Yes, both are required for now. We plan to refactor this in the
        future.
-   Q: Isn't the inheritance of the Test classes just migrated to inheritance
    of TestRules?
    -   A: Yes. During the migration, we plan to maintain a 1:1 mapping between
        the test base classes and TestRules (e.g. ContentShellTestBase to
        ContentShellTestRule in this
        [CL](https://codereview.chromium.org/2632043002)).
        This allows the auto convert script to replace API calls in any
        JUnit3 tests. After the migration, we plan to refactor the TestRules to
        be more modular.

If you have any other questions, feel free to report in [this bug][7].

## Links and Crbugs

- [Android Test Support Library documentation][1]
- [Auto change script][5]
- [Crbug for JUnit3 to JUnit4 migration][7]

[1]: https://developer.android.com/topic/libraries/testing-support-library/index.html
[2]: https://cs.chromium.org/chromium/src/android_webview/tools/system_webview_shell/layout_tests/AndroidManifest.xml?l=36
[3]: http://junit.org/junit4/javadoc/4.12/org/junit/rules/TestRule.html
[4]: https://developer.android.com/reference/android/support/test/rule/ActivityTestRule.html
[5]: https://github.com/yoland68/chromium-junit-auto-migrate
[6]: http://github.com/skyisle/android-test-kit/issues/121
[7]: https://bugs.chromium.org/p/chromium/issues/detail?id=640116
[8]: http://junit.org/junit4/javadoc/4.12/org/junit/rules/RuleChain.html
[9]: https://developer.android.com/reference/android/app/Instrumentation.html#runOnMainSync(java.lang.Runnable)
[10]: https://developer.android.com/reference/android/support/test/rule/UiThreadTestRule.html#runOnUiThread(java.lang.Runnable)
[11]: /chrome/test/android/javatests/src/org/chromium/chrome/test/BottomSheetTestRule.java
[12]: https://bugs.chromium.org/p/chromium/issues/detail?id=734553
[13]: /base/test/android/javatests/src/org/chromium/base/test/util/CommandLineFlags.java