Avro
  1. Avro
  2. AVRO-1313

Java: Add system property for disabling sun.misc.Unsafe

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.7.5
    • Component/s: None
    • Labels:
      None

      Description

      We should be able to disable use of sun.misc.Unsafe.

      I propose that if the system property "avro.disable.unsafe" is non-null, we use reflection rather than Unsafe.

      1. AVRO-1313-v2.patch
        2 kB
        Scott Carey
      2. AVRO-1313.patch
        1 kB
        Scott Carey

        Activity

        Hide
        Scott Carey added a comment -

        This patch adds the check for "avro.disable.unsafe".

        When -Davro.disable.unsafe is added to the command line, performance drops as expected for field access, but array performance is still fast:

        Unsafe On:

                                                           test name     time    M entries/sec   M bytes/sec  bytes/cycle
                                 ReflectRecordRead:   7405 ms       2.251        87.343        808498
                                ReflectRecordWrite:   4786 ms       3.482       135.121        808498
                              ReflectBigRecordRead:   7478 ms       1.337        82.089        767380
                             ReflectBigRecordWrite:   4984 ms       2.006       123.153        767380
                                  ReflectFloatRead:   6927 ms       0.000       115.486       1000004
                                 ReflectFloatWrite:   1087 ms       0.001       735.371       1000004
                                 ReflectDoubleRead:   8678 ms       0.000       184.369       2000004
                                ReflectDoubleWrite:   2398 ms       0.000       666.980       2000004
                               ReflectIntArrayRead:  11756 ms       1.418        58.503        859709
                              ReflectIntArrayWrite:   3798 ms       4.388       181.070        859709
                              ReflectLongArrayRead:   6542 ms       1.274        98.481        805344
                             ReflectLongArrayWrite:   2189 ms       3.806       294.278        805344
                            ReflectDoubleArrayRead:   6316 ms       1.583       103.625        818144
                           ReflectDoubleArrayWrite:   1589 ms       6.292       411.827        818144
                             ReflectFloatArrayRead:  13986 ms       1.430        48.400        846172
                            ReflectFloatArrayWrite:   2953 ms       6.771       229.186        846172
                       ReflectNestedFloatArrayRead:  16618 ms       1.203        40.733        846172
                      ReflectNestedFloatArrayWrite:   4841 ms       4.131       139.820        846172
                      ReflectNestedObjectArrayRead:  12905 ms       0.310        39.989        645104
                     ReflectNestedObjectArrayWrite:   6868 ms       0.582        75.139        645104
                  ReflectNestedLargeFloatArrayRead:  10141 ms       0.329        85.781       1087381
                 ReflectNestedLargeFloatArrayWrite:   2049 ms       1.626       424.432       1087381
           ReflectNestedLargeFloatArrayBlockedRead:  10501 ms       0.317        83.899       1101357
          ReflectNestedLargeFloatArrayBlockedWrite:   5554 ms       0.600       158.634       1101357
        

        Unsafe Off:

                                                           test name     time    M entries/sec   M bytes/sec  bytes/cycle
                                 ReflectRecordRead:  13282 ms       1.255        48.694        808498
                                ReflectRecordWrite:   8981 ms       1.856        72.011        808498
                              ReflectBigRecordRead:  17118 ms       0.584        35.863        767380
                             ReflectBigRecordWrite:  13178 ms       0.759        46.584        767380
                                  ReflectFloatRead:   6713 ms       0.000       119.160       1000004
                                 ReflectFloatWrite:   2444 ms       0.000       327.229       1000004
                                 ReflectDoubleRead:   8094 ms       0.000       197.677       2000004
                                ReflectDoubleWrite:   2133 ms       0.000       749.844       2000004
                               ReflectIntArrayRead:  12127 ms       1.374        56.712        859709
                              ReflectIntArrayWrite:   3832 ms       4.349       179.463        859709
                              ReflectLongArrayRead:   6312 ms       1.320       102.059        805344
                             ReflectLongArrayWrite:   2548 ms       3.269       252.785        805344
                            ReflectDoubleArrayRead:   7460 ms       1.340        87.726        818144
                           ReflectDoubleArrayWrite:   2048 ms       4.882       319.526        818144
                             ReflectFloatArrayRead:  11761 ms       1.700        57.554        846172
                            ReflectFloatArrayWrite:   3370 ms       5.935       200.871        846172
                       ReflectNestedFloatArrayRead:  15946 ms       1.254        42.450        846172
                      ReflectNestedFloatArrayWrite:   6429 ms       3.111       105.291        846172
                      ReflectNestedObjectArrayRead:  17478 ms       0.229        29.527        645104
                     ReflectNestedObjectArrayWrite:  12148 ms       0.329        42.480        645104
                  ReflectNestedLargeFloatArrayRead:   9012 ms       0.370        96.524       1087381
                 ReflectNestedLargeFloatArrayWrite:   3950 ms       0.844       220.223       1087381
           ReflectNestedLargeFloatArrayBlockedRead:  10390 ms       0.321        84.795       1101357
          ReflectNestedLargeFloatArrayBlockedWrite:   6972 ms       0.478       126.368       1101357
        

        This also fixes a bug introduced in AVRO-1282 for reflection where non-public classes/fields could not be read or written when using reflection, but could with Unsafe.

        I will commit this soon if there are no objections.

        Show
        Scott Carey added a comment - This patch adds the check for "avro.disable.unsafe". When -Davro.disable.unsafe is added to the command line, performance drops as expected for field access, but array performance is still fast: Unsafe On: test name time M entries/sec M bytes/sec bytes/cycle ReflectRecordRead: 7405 ms 2.251 87.343 808498 ReflectRecordWrite: 4786 ms 3.482 135.121 808498 ReflectBigRecordRead: 7478 ms 1.337 82.089 767380 ReflectBigRecordWrite: 4984 ms 2.006 123.153 767380 ReflectFloatRead: 6927 ms 0.000 115.486 1000004 ReflectFloatWrite: 1087 ms 0.001 735.371 1000004 ReflectDoubleRead: 8678 ms 0.000 184.369 2000004 ReflectDoubleWrite: 2398 ms 0.000 666.980 2000004 ReflectIntArrayRead: 11756 ms 1.418 58.503 859709 ReflectIntArrayWrite: 3798 ms 4.388 181.070 859709 ReflectLongArrayRead: 6542 ms 1.274 98.481 805344 ReflectLongArrayWrite: 2189 ms 3.806 294.278 805344 ReflectDoubleArrayRead: 6316 ms 1.583 103.625 818144 ReflectDoubleArrayWrite: 1589 ms 6.292 411.827 818144 ReflectFloatArrayRead: 13986 ms 1.430 48.400 846172 ReflectFloatArrayWrite: 2953 ms 6.771 229.186 846172 ReflectNestedFloatArrayRead: 16618 ms 1.203 40.733 846172 ReflectNestedFloatArrayWrite: 4841 ms 4.131 139.820 846172 ReflectNestedObjectArrayRead: 12905 ms 0.310 39.989 645104 ReflectNestedObjectArrayWrite: 6868 ms 0.582 75.139 645104 ReflectNestedLargeFloatArrayRead: 10141 ms 0.329 85.781 1087381 ReflectNestedLargeFloatArrayWrite: 2049 ms 1.626 424.432 1087381 ReflectNestedLargeFloatArrayBlockedRead: 10501 ms 0.317 83.899 1101357 ReflectNestedLargeFloatArrayBlockedWrite: 5554 ms 0.600 158.634 1101357 Unsafe Off: test name time M entries/sec M bytes/sec bytes/cycle ReflectRecordRead: 13282 ms 1.255 48.694 808498 ReflectRecordWrite: 8981 ms 1.856 72.011 808498 ReflectBigRecordRead: 17118 ms 0.584 35.863 767380 ReflectBigRecordWrite: 13178 ms 0.759 46.584 767380 ReflectFloatRead: 6713 ms 0.000 119.160 1000004 ReflectFloatWrite: 2444 ms 0.000 327.229 1000004 ReflectDoubleRead: 8094 ms 0.000 197.677 2000004 ReflectDoubleWrite: 2133 ms 0.000 749.844 2000004 ReflectIntArrayRead: 12127 ms 1.374 56.712 859709 ReflectIntArrayWrite: 3832 ms 4.349 179.463 859709 ReflectLongArrayRead: 6312 ms 1.320 102.059 805344 ReflectLongArrayWrite: 2548 ms 3.269 252.785 805344 ReflectDoubleArrayRead: 7460 ms 1.340 87.726 818144 ReflectDoubleArrayWrite: 2048 ms 4.882 319.526 818144 ReflectFloatArrayRead: 11761 ms 1.700 57.554 846172 ReflectFloatArrayWrite: 3370 ms 5.935 200.871 846172 ReflectNestedFloatArrayRead: 15946 ms 1.254 42.450 846172 ReflectNestedFloatArrayWrite: 6429 ms 3.111 105.291 846172 ReflectNestedObjectArrayRead: 17478 ms 0.229 29.527 645104 ReflectNestedObjectArrayWrite: 12148 ms 0.329 42.480 645104 ReflectNestedLargeFloatArrayRead: 9012 ms 0.370 96.524 1087381 ReflectNestedLargeFloatArrayWrite: 3950 ms 0.844 220.223 1087381 ReflectNestedLargeFloatArrayBlockedRead: 10390 ms 0.321 84.795 1101357 ReflectNestedLargeFloatArrayBlockedWrite: 6972 ms 0.478 126.368 1101357 This also fixes a bug introduced in AVRO-1282 for reflection where non-public classes/fields could not be read or written when using reflection, but could with Unsafe. I will commit this soon if there are no objections.
        Hide
        Scott Carey added a comment -

        Ready for review.

        Show
        Scott Carey added a comment - Ready for review.
        Hide
        Doug Cutting added a comment -

        Should we add a reflect test with a private field?

        Show
        Doug Cutting added a comment - Should we add a reflect test with a private field?
        Hide
        Scott Carey added a comment - - edited

        Even better, the check at runtime when loading should check that it will work.

        This patch (-v2) changes the runtime test to have fields of all types and validates that it works with all of them before loading an implementation.

        Both Unsafe and Reflect cases of these code paths are covered in the Unit tests.

        Show
        Scott Carey added a comment - - edited Even better, the check at runtime when loading should check that it will work. This patch (-v2) changes the runtime test to have fields of all types and validates that it works with all of them before loading an implementation. Both Unsafe and Reflect cases of these code paths are covered in the Unit tests.
        Hide
        Doug Cutting added a comment -

        +1

        Show
        Doug Cutting added a comment - +1
        Hide
        Scott Carey added a comment -

        Committed in revision 1478244.

        Show
        Scott Carey added a comment - Committed in revision 1478244.
        Hide
        Hudson added a comment -

        Integrated in AvroJava #370 (See https://builds.apache.org/job/AvroJava/370/)
        AVRO-1313. Java: Add system property avro.disable.unsafe for disabling
        use of sun.misc.Unsafe. (Revision 1478244)

        Result = SUCCESS
        scottcarey :
        Files :

        • /avro/trunk/CHANGES.txt
        • /avro/trunk/lang/java/avro/src/main/java/org/apache/avro/reflect/FieldAccessReflect.java
        • /avro/trunk/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectionUtil.java
        Show
        Hudson added a comment - Integrated in AvroJava #370 (See https://builds.apache.org/job/AvroJava/370/ ) AVRO-1313 . Java: Add system property avro.disable.unsafe for disabling use of sun.misc.Unsafe. (Revision 1478244) Result = SUCCESS scottcarey : Files : /avro/trunk/CHANGES.txt /avro/trunk/lang/java/avro/src/main/java/org/apache/avro/reflect/FieldAccessReflect.java /avro/trunk/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectionUtil.java

          People

          • Assignee:
            Scott Carey
            Reporter:
            Scott Carey
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development