Skip to content

Android: Move JNI keyboard/clipboard handling into backend, simplify example (#3446) - #9469

Open
Turtle-PB wants to merge 21 commits into
ocornut:masterfrom
Turtle-PB:enhance/android-backend-cleanup
Open

Android: Move JNI keyboard/clipboard handling into backend, simplify example (#3446)#9469
Turtle-PB wants to merge 21 commits into
ocornut:masterfrom
Turtle-PB:enhance/android-backend-cleanup

Conversation

@Turtle-PB

@Turtle-PB Turtle-PB commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Moves all JNI boilerplate (soft keyboard, Unicode char polling, clipboard) from the example into the imgui_impl_android backend, and adds native sensor + display metrics support. The example stays clean and showcases the device's full capabilities.

Addresses the long-standing FIXMEs:

  • #3446: On-screen keyboard / Unicode input handled by the app
  • Missing clipboard support
  • No sensor access
  • No display metrics (DPI/density)

Backend API additions

// Init now accepts optional asset_manager and native_activity
bool ImGui_ImplAndroid_Init(ANativeWindow* window, AAssetManager* asset_manager = nullptr, jobject native_activity = nullptr);

// Keyboard (auto-handled in NewFrame, also explicit)
void ImGui_ImplAndroid_ShowSoftKeyboard();
void ImGui_ImplAndroid_HideSoftKeyboard();

// Sensors — NDK ASensor API, no JNI
bool     ImGui_ImplAndroid_EnableSensor(int sensor_type);
void     ImGui_ImplAndroid_DisableSensor(int sensor_type);
void     ImGui_ImplAndroid_GetSensorData(int sensor_type, ImGui_ImplAndroid_SensorData* out_data);
bool     ImGui_ImplAndroid_IsSensorAvailable(int sensor_type);

// Display metrics — DPI, density, refresh rate, orientation
void     ImGui_ImplAndroid_GetDisplayMetrics(ImGui_ImplAndroid_DisplayMetrics* out_metrics);

What moved into the backend

  • Soft keyboard: backend checks io.WantTextInput in NewFrame(), calls activity's showSoftInput()/hideSoftInput() via JNI
  • Unicode polling: backend polls pollUnicodeChar() via JNI in NewFrame(), feeds to io.AddInputCharacter()
  • Clipboard: backend uses JNI to call ClipboardManager.setText()/getText(), wires io.SetClipboardTextFn/GetClipboardTextFn
  • Sensors: backend uses NDK ASensorManager/ASensorEventQueue to drain events in NewFrame() — no JNI needed
  • Display metrics: backend queries DisplayMetrics + Display via JNI in Init() for DPI, density, xdpi/ydpi, refresh rate, orientation

Example changes

The example Init() is now a single clean call:

ImGui_ImplAndroid_Init(g_App->window, app->activity->assetManager, app->activity->clazz);

The example showcases:

  • Sensor window: live accelerometer, gyroscope, magnetometer, light, proximity with visual bars
  • Display window: resolution, DPI, density, refresh rate, orientation
  • Auto-scaled style using actual device density (not hardcoded 2.0f)

Removed ~120 lines of JNI boilerplate. Zero JNI calls in main.cpp.

Kotlin activity

Kept 3 JNI-bridge methods (showSoftInput/hideSoftInput/pollUnicodeChar) — Android's native API does not provide these. They are called by the backend, not by app code.

No binaries

Per feedback on #9467, zero binary files added. Only source files changed.

Files changed

File Change
backends/imgui_impl_android.h +81 lines: sensor API, display API, feature list
backends/imgui_impl_android.cpp +536 lines: JNI helpers, sensor queue, display metrics, clipboard
examples/.../main.cpp Clean example + sensor/display windows
examples/.../CMakeLists.txt Added sensor to link libs
examples/.../MainActivity.kt Cleaned up, documented

Addresses #3446.

…lay metrics (ocornut#3446)

Backend improvements (imgui_impl_android.h/.cpp):
- Moved JNI soft-keyboard and Unicode char polling from the example into the backend.
  The backend automatically shows/hides the soft keyboard based on io.WantTextInput
  during NewFrame().
- Moved JNI Unicode character polling from the example into the backend.
  Characters are polled and fed to io.AddInputCharacter() automatically.
- Added clipboard support via JNI to Android ClipboardManager.
  io.SetClipboardTextFn / io.GetClipboardTextFn are now set by the backend.
- Added ImGui_ImplAndroid_ShowSoftKeyboard() / HideSoftKeyboard() public API.
- Init() now accepts optional asset_manager and native_activity params.
  When native_activity is provided, JNI features are enabled automatically.
  When nullptr, the backend degrades gracefully (no keyboard/clipboard).
- Added sensor support via NDK ASensor API (no JNI needed):
  Accelerometer, Gyroscope, Magnetometer, Light, Proximity, Pressure,
  Humidity, Ambient Temperature.
  API: EnableSensor(), DisableSensor(), GetSensorData(), IsSensorAvailable().
  Sensor events are drained non-blocking in NewFrame().
- Added display metrics via JNI:
  DPI, density, xdpi/ydpi, refresh rate, orientation, resolution.
  API: GetDisplayMetrics().
  Used to auto-scale ImGui style to the device's actual density.

Example cleanup (main.cpp):
- Removed ~120 lines of JNI boilerplate (ShowSoftKeyboardInput, PollUnicodeChars,
  GetAssetData). The example is now a clean Init -> Loop -> Render -> Shutdown.
- Single ImGui_ImplAndroid_Init() call passes the activity object; the backend
  handles the rest.
- Added sensor window: live accelerometer, gyroscope, magnetometer, light, proximity
  readouts with visual progress bars.
- Added display window: resolution, DPI, density, refresh rate, orientation.
- Uses actual device density for style scaling instead of hardcoded 2.0f.

CMakeLists.txt: added 'sensor' to target_link_libraries for ASensor API.

Kotlin activity (MainActivity.kt):
- Kept the three JNI-bridge methods (showSoftInput/hideSoftInput/pollUnicodeChar)
  since Android's native API does not provide these. Cleaned up and documented
  that they are called by the backend, not by app code.
@Turtle-PB
Turtle-PB force-pushed the enhance/android-backend-cleanup branch from 084847e to 2f36763 Compare July 13, 2026 17:20
@ocornut

ocornut commented Jul 15, 2026

Copy link
Copy Markdown
Owner

Thank you for your PR!

This looks good overall but I'll need to address several details.

Example:

  • There was supposedly a fix in the destroyRequested handling in main.cpp which should be explicitly documented.
  • Why the EGL init block changes?
  • Comments have been changed. They should follow the exact format of the 32 other examples. So stuff like // Setup Dear ImGui context -> // --- Dear ImGui setup --- should be reversed. Most other removed comments should be reversed. And GetAssetData() ? Honestly it is pretty worrying that you removed so many things, it lowers the general amount of trust-for-care for an otherwise seemingly great PR. Likewise for a few standard comments in imgui_impl_android.cpp (in doubt always diff the whole thing).

Adding display metrics I am ok with because needed to setup main scale.
HOWEVER, the addition of sensor stuff seems completely off-topic to dear imgui and arbitrary.
Please keep it as a separate commit + provide a screenshot.

@Turtle-PB

Turtle-PB commented Jul 15, 2026 via email

Copy link
Copy Markdown
Contributor Author

Turtle-PB and others added 12 commits July 15, 2026 17:36
…IniSettingsFromMemory().

While not done with core, this would be useful to load .ini over a clean slate.
…d unnecessary InputTextDeactivateHook() call on manual deactivation. InputTextDeactivateHook() only takes a record when Edited + callback marks edited. (ocornut#9476, ocornut#701)

for _NoLiveEdit it's easier than we don't use IsItemDeactivatedAfterEdit() in InputText()'s `if (g.InputTextDeactivatedState.ID == id)` block.
@tope99

tope99 commented Jul 19, 2026

Copy link
Copy Markdown

I have been using ImGui on Android in few pet projects. There were quite a few changes which needed to be implemented to make it usable and I tried to include all these in code generated by ImRAD. I feel like this PR tries to address some issues but the result may not be customizable enough to cover all cases needed. At the same time some other important issues are not addressed. I think reaching a broader consensus and testing should to be done before merging it.

My observations:

  • PR queries display rotation but only in ImplAndroid_Init? What if rotation changes afterwards? If rotation is handled it should be handled in full

  • Showing/hiding kbd is a big topic and I don't think moving it into the backend is a good idea. At least in its current form.

    • There are multiple types of keyboard types f.e. keyboard only for a number input. ImRAD generates code which switches keyboard type based on the Input widget settings. There is also a keyboard action button which is customizable. These things are not possible with this PR.

    • Keyboards in android react differently. For example when the user touches the screen and scrolls the keyboard won't hide. When a button is pressed it won't hide until the touch is released. It's important because when keyboard is suddenly hidden the position of widgets may change when they are f.e. anchored to the bottom. I solved this by resubmitting WantTextInput and by hiding the keyboard only when touch is released. This PR makes it impossible when ImplAndroid_NewFrame is called.

    • Ideally the keyboard should be made hidden when user presses the back button. That is regardless of WantTextInput as this is normal android behavior. This is not addressed.

  • dispatchKeyEvent never worked well for me. Some keys were always ignored so I am using it only for KeyEvent.KEYCODE_BACK. For other keys I have to use TextWatcher.OnInputCharacter. Then for keyboard action button TextView.OnEditorActionListener was needed. That is already a problem with current version of android example but when it is moved into the backed it only makes it harder to reimplement.

  • GetAssetData and is an useful function to load assets. I am not sure why it was removed together with the font loading section.

Other issues with main.cpp which are already present in the current android example.

  • crash when switching apps. I am not an android expert but for me handling APP_CMD_TERM_WINDOW differently fixed it (only release EGL surface, create new surface when switched back).

  • querying NavBar height and bottom area height should be supported as well so the app can render to the correct display area. But keep it configurable as some apps prefer to render over bottom bar buttons to maximize content display and some don't.

  • support long press event with haptic feedback.

@Turtle-PB

Copy link
Copy Markdown
Contributor Author

I will address the issue on my fork if I can identify a fix I will let you know. I need to stay with the community guidelines and standards without over complicating things.

ocornut#3446)

Backend improvements (imgui_impl_android.h/.cpp):
- Moved JNI soft-keyboard and Unicode char polling from the example into
  the backend. The application no longer needs any JNI boilerplate.
- Added clipboard support via JNI to Android ClipboardManager. (ocornut#7259)
- Added display metrics (DPI, density, refresh rate, orientation) via JNI.
  Used by the example to set main_scale from actual device density instead
  of hardcoded 2.0f.
- BREAKING CHANGE: ImGui_ImplAndroid_Init() now takes additional
  asset_manager and native_activity parameters. See before/after in header.

Example (main.cpp):
- Removed ShowSoftKeyboardInput() and PollUnicodeChars() JNI helpers (~120 lines).
- Removed their forward declarations and per-frame calls.
- Updated Init() call to new signature.
- Use ImGui_ImplAndroid_GetDisplayMetrics() for main_scale.
- All other comments, EGL block, GetAssetData, APP_CMD cases unchanged.

MainActivity.kt: unchanged (same 3 JNI bridge methods, now called by backend).

docs/CHANGELOG.txt: Added entries for backend and example changes.
CRITICAL FIX: Restore NewFrame/Shutdown/GetDisplayMetrics lost in rebase
- File was truncated at line 605 during rebase onto upstream/master
- Restored all three functions with proper implementations

REVIEW FEEDBACK ADDRESSED:
1. Rotation: Re-query display metrics in NewFrame when window resizes
2. Keyboard opt-in: SetJniEnabled(false by default) - app retains control
   - SetKeyboardType() / SetKeyboardAction() for customization
3. Keyboard timing: GetWantTextInput() / ResubmitTextInput() for app control
4. Back button: Hides soft keyboard when visible (consumes event)
5. Unicode input: Kept as optional JNI plumbing (app decides when to poll)
6. GetAssetData: Documented in PR_9469_REVIEW_RESPONSE.md (restore in main.cpp)
7. TERM_WINDOW: Documented (EGL surface lifecycle fix in main.cpp)
8. NavBar/insets: GetNavBarHeight() + GetBottomInset() (DisplayCutout API 28+)
   Long press: SetLongPressCallback() + SetLongPressDuration()
   Pressure: Full pressure sensitivity system

PRESSURE SENSITIVITY:
- Per-pointer pressure tracking (up to 10 pointers)
- GetTouchPressure(pointer_id) API
- SetPressureEnabled() / SetPressureThreshold()
- Long-press requires sustained min pressure (not just timer)
- Pressure-weighted scrolling (firmer press = faster scroll)
- Move events update pressure and cancel long-press if pressure drops

All APIs are opt-in. Default behavior preserves pre-PR ocornut#9469 app control.
Turtle-PB added a commit to Turtle-PB/imgui_xl that referenced this pull request Jul 19, 2026
- Android backend: Address ImRAD review feedback on PR ocornut#9469
- Restore NewFrame/Shutdown/GetDisplayMetrics lost in rebase
- Add opt-in JNI features, pressure sensitivity, NavBar/insets, long-press
- Add PR_9469_REVIEW_RESPONSE.md documenting all 8 review points
@ocornut

ocornut commented Jul 19, 2026

Copy link
Copy Markdown
Owner

Thank you @tope99 for your feedback, much appreciated. Could the PR be constructed to have 1 feature per commit so we can more easily merge individual things when there’s an agreement?.

I agree that facilitating code (eg integrating osd keyboard in backend) should not remove agency for the app to do advanced things. So this needs to be designed accordingly.

Turtle-PB added a commit to Turtle-PB/imgui_xl that referenced this pull request Jul 20, 2026
- Hero badges: '3 PRs merged' + '3 PRs open' + 'Audio + MIDI + GUI Builder'
- Stats: 6 upstream PRs, 3 merged, 12 widget types, 2 royalty-free tracks
- PR list: All 6 PRs with correct status dots (merged=purple, open=green)
  - ocornut#9467 Android CI (merged)
  - ocornut#9473 OpenGL GL_UNPACK (merged)
  - ocornut#9474 Android touch + SDL2 (merged)
  - ocornut#9481 Texture destroy deferral (open)
  - ocornut#9483 CBDT/COLR emoji fonts (open)
  - ocornut#9469 Android backend + pressure (open)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants