-
-
Notifications
You must be signed in to change notification settings - Fork 465
feat: merge tombstone and native sdk events #5037
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
supervacuus
wants to merge
9
commits into
main
Choose a base branch
from
feat/tombstone_native_sdk_merge
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+404
−9
Draft
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
028f17b
feat: merge tombstone and native sdk events
supervacuus 6befa43
add preliminary change log
supervacuus aa8f65f
Merge branch 'main' into feat/tombstone_native_sdk_merge
supervacuus bf14914
Merge branch 'main' into feat/tombstone_native_sdk_merge
supervacuus d7d5447
add preliminary change log
supervacuus ba1bfc7
apply review+sync feedback
supervacuus 7d73c41
Merge branch 'main' into feat/tombstone_native_sdk_merge
supervacuus cd970cb
add tombstone manifest flag
supervacuus b832e7c
remove tombstone-native correlation via processStateSummary
supervacuus File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| .DS_Store | ||
| .java-version | ||
| .idea/ | ||
| .gradle/ | ||
| .run/ | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
247 changes: 247 additions & 0 deletions
247
sentry-android-core/src/main/java/io/sentry/android/core/NativeEventCollector.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,247 @@ | ||
| package io.sentry.android.core; | ||
|
|
||
| import static io.sentry.cache.EnvelopeCache.PREFIX_CURRENT_SESSION_FILE; | ||
| import static io.sentry.cache.EnvelopeCache.PREFIX_PREVIOUS_SESSION_FILE; | ||
| import static io.sentry.cache.EnvelopeCache.STARTUP_CRASH_MARKER_FILE; | ||
|
|
||
| import io.sentry.SentryEnvelope; | ||
| import io.sentry.SentryEnvelopeItem; | ||
| import io.sentry.SentryEvent; | ||
| import io.sentry.SentryItemType; | ||
| import io.sentry.SentryLevel; | ||
| import java.io.BufferedInputStream; | ||
| import java.io.BufferedReader; | ||
| import java.io.ByteArrayInputStream; | ||
| import java.io.File; | ||
| import java.io.FileInputStream; | ||
| import java.io.InputStream; | ||
| import java.io.InputStreamReader; | ||
| import java.io.Reader; | ||
| import java.nio.charset.StandardCharsets; | ||
| import java.util.ArrayList; | ||
| import java.util.Date; | ||
| import java.util.List; | ||
| import org.jetbrains.annotations.ApiStatus; | ||
| import org.jetbrains.annotations.NotNull; | ||
| import org.jetbrains.annotations.Nullable; | ||
|
|
||
| /** | ||
| * Collects native crash events from the outbox directory. These events can be correlated with | ||
| * tombstone events from ApplicationExitInfo to avoid sending duplicate crash reports. | ||
| */ | ||
| @ApiStatus.Internal | ||
| public final class NativeEventCollector { | ||
|
|
||
| private static final String NATIVE_PLATFORM = "native"; | ||
|
|
||
| // TODO: will be replaced with the correlationId once the Native SDK supports it | ||
| private static final long TIMESTAMP_TOLERANCE_MS = 5000; | ||
|
|
||
| private final @NotNull SentryAndroidOptions options; | ||
| private final @NotNull List<NativeEventData> nativeEvents = new ArrayList<>(); | ||
| private boolean collected = false; | ||
|
|
||
| public NativeEventCollector(final @NotNull SentryAndroidOptions options) { | ||
| this.options = options; | ||
| } | ||
|
|
||
| /** Holds a native event along with its source file for later deletion. */ | ||
| public static final class NativeEventData { | ||
| private final @NotNull SentryEvent event; | ||
| private final @NotNull File file; | ||
| private final @NotNull SentryEnvelope envelope; | ||
| private final long timestampMs; | ||
|
|
||
| NativeEventData( | ||
| final @NotNull SentryEvent event, | ||
| final @NotNull File file, | ||
| final @NotNull SentryEnvelope envelope, | ||
| final long timestampMs) { | ||
| this.event = event; | ||
| this.file = file; | ||
| this.envelope = envelope; | ||
| this.timestampMs = timestampMs; | ||
| } | ||
|
|
||
| public @NotNull SentryEvent getEvent() { | ||
| return event; | ||
| } | ||
|
|
||
| public @NotNull File getFile() { | ||
| return file; | ||
| } | ||
|
|
||
| public @NotNull SentryEnvelope getEnvelope() { | ||
| return envelope; | ||
| } | ||
|
|
||
| public long getTimestampMs() { | ||
| return timestampMs; | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Scans the outbox directory and collects all native crash events. This method should be called | ||
| * once before processing tombstones. Subsequent calls are no-ops. | ||
| */ | ||
| public void collect() { | ||
| if (collected) { | ||
| return; | ||
| } | ||
| collected = true; | ||
|
|
||
| final @Nullable String outboxPath = options.getOutboxPath(); | ||
| if (outboxPath == null) { | ||
| options | ||
| .getLogger() | ||
| .log(SentryLevel.DEBUG, "Outbox path is null, skipping native event collection."); | ||
| return; | ||
| } | ||
|
|
||
| final File outboxDir = new File(outboxPath); | ||
| if (!outboxDir.isDirectory()) { | ||
| options.getLogger().log(SentryLevel.DEBUG, "Outbox path is not a directory: %s", outboxPath); | ||
| return; | ||
| } | ||
|
|
||
| final File[] files = outboxDir.listFiles((d, name) -> isRelevantFileName(name)); | ||
| if (files == null || files.length == 0) { | ||
| options.getLogger().log(SentryLevel.DEBUG, "No envelope files found in outbox."); | ||
| return; | ||
| } | ||
|
|
||
| options | ||
| .getLogger() | ||
| .log(SentryLevel.DEBUG, "Scanning %d files in outbox for native events.", files.length); | ||
|
|
||
| for (final File file : files) { | ||
| if (!file.isFile()) { | ||
| continue; | ||
| } | ||
|
|
||
| final @Nullable NativeEventData nativeEventData = extractNativeEventFromFile(file); | ||
| if (nativeEventData != null) { | ||
| nativeEvents.add(nativeEventData); | ||
| options | ||
| .getLogger() | ||
| .log( | ||
| SentryLevel.DEBUG, | ||
| "Found native event in outbox: %s (timestamp: %d)", | ||
| file.getName(), | ||
| nativeEventData.getTimestampMs()); | ||
| } | ||
| } | ||
|
|
||
| options | ||
| .getLogger() | ||
| .log(SentryLevel.DEBUG, "Collected %d native events from outbox.", nativeEvents.size()); | ||
| } | ||
|
|
||
| /** | ||
| * Finds a native event that matches the given tombstone timestamp. If a match is found, it is | ||
| * removed from the internal list so it won't be matched again. | ||
| * | ||
| * <p>This method will lazily collect native events from the outbox on first call. | ||
| * | ||
| * @param tombstoneTimestampMs the timestamp from ApplicationExitInfo | ||
| * @return the matching native event data, or null if no match found | ||
| */ | ||
| public @Nullable NativeEventData findAndRemoveMatchingNativeEvent( | ||
| final long tombstoneTimestampMs) { | ||
|
|
||
| // Lazily collect on first use (runs on executor thread, not main thread) | ||
| collect(); | ||
|
|
||
| for (final NativeEventData nativeEvent : nativeEvents) { | ||
| final long timeDiff = Math.abs(tombstoneTimestampMs - nativeEvent.getTimestampMs()); | ||
| if (timeDiff <= TIMESTAMP_TOLERANCE_MS) { | ||
| options | ||
| .getLogger() | ||
| .log(SentryLevel.DEBUG, "Matched native event by timestamp (diff: %d ms)", timeDiff); | ||
| nativeEvents.remove(nativeEvent); | ||
| return nativeEvent; | ||
| } | ||
| } | ||
|
|
||
| return null; | ||
| } | ||
|
|
||
| /** | ||
| * Deletes a native event file from the outbox. | ||
| * | ||
| * @param nativeEventData the native event data containing the file reference | ||
| * @return true if the file was deleted successfully | ||
| */ | ||
| public boolean deleteNativeEventFile(final @NotNull NativeEventData nativeEventData) { | ||
| final File file = nativeEventData.getFile(); | ||
| try { | ||
| if (file.delete()) { | ||
| options | ||
| .getLogger() | ||
| .log(SentryLevel.DEBUG, "Deleted native event file from outbox: %s", file.getName()); | ||
| return true; | ||
| } else { | ||
| options | ||
| .getLogger() | ||
| .log( | ||
| SentryLevel.WARNING, | ||
| "Failed to delete native event file: %s", | ||
| file.getAbsolutePath()); | ||
| return false; | ||
| } | ||
| } catch (Throwable e) { | ||
| options | ||
| .getLogger() | ||
| .log( | ||
| SentryLevel.ERROR, e, "Error deleting native event file: %s", file.getAbsolutePath()); | ||
| return false; | ||
| } | ||
| } | ||
|
|
||
| private @Nullable NativeEventData extractNativeEventFromFile(final @NotNull File file) { | ||
| try (final InputStream stream = new BufferedInputStream(new FileInputStream(file))) { | ||
| final SentryEnvelope envelope = options.getEnvelopeReader().read(stream); | ||
| if (envelope == null) { | ||
| return null; | ||
| } | ||
|
|
||
| for (final SentryEnvelopeItem item : envelope.getItems()) { | ||
| if (!SentryItemType.Event.equals(item.getHeader().getType())) { | ||
| continue; | ||
| } | ||
|
|
||
| try (final Reader eventReader = | ||
| new BufferedReader( | ||
| new InputStreamReader( | ||
| new ByteArrayInputStream(item.getData()), StandardCharsets.UTF_8))) { | ||
| final SentryEvent event = | ||
| options.getSerializer().deserialize(eventReader, SentryEvent.class); | ||
| if (event != null && NATIVE_PLATFORM.equals(event.getPlatform())) { | ||
| final long timestampMs = extractTimestampMs(event); | ||
| return new NativeEventData(event, file, envelope, timestampMs); | ||
| } | ||
| } | ||
| } | ||
| } catch (Throwable e) { | ||
| options | ||
| .getLogger() | ||
| .log(SentryLevel.DEBUG, e, "Error reading envelope file: %s", file.getAbsolutePath()); | ||
| } | ||
| return null; | ||
| } | ||
|
|
||
| private long extractTimestampMs(final @NotNull SentryEvent event) { | ||
| final @Nullable Date timestamp = event.getTimestamp(); | ||
| if (timestamp != null) { | ||
| return timestamp.getTime(); | ||
| } | ||
| return 0; | ||
| } | ||
|
|
||
| private boolean isRelevantFileName(final @Nullable String fileName) { | ||
| return fileName != null | ||
| && !fileName.startsWith(PREFIX_CURRENT_SESSION_FILE) | ||
| && !fileName.startsWith(PREFIX_PREVIOUS_SESSION_FILE) | ||
| && !fileName.startsWith(STARTUP_CRASH_MARKER_FILE); | ||
| } | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm a bit worried of ending up with too many envelopes in memory here and potentially running into OOMs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you tell me which code in particular is likely to cause references to spill? I'm not entirely sure. My assumption was that there would typically be very few native events (on average, 1 after each native crash), since they are usually sent on the next restart (and we only collect native events and their envelopes). So I would guess that, while iterating over and deserializing all envelopes, each iteration should leave the discarded ones ready for GC, right?
My bigger worry was the attachments that we pass around with the envelopes (yes, they are currently not attached to the tombstone events). I am fine with not passing around the envelope for the event, but we must ensure it is loaded at some point. Was this your concern? We could store only the envelope path in
NativeEventDataand only load it once we have a matching tombstone. Or did I completely misunderstand?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As far as I can see we
extractNativeEventFromFile()parses any envelope, so there could be transactions, or multiple crashes, e.g. if the device was offline. Or e.g. the app continuously crashes (in native code) during startup, a fix get released and now all unset envelopes pile up.So I would just be a bit more defensive here, e.g. drop transactions, and have a sane maximum for
nativeEvents.size()There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would love to be defensive; I would like to discard envelopes based solely on the filename (or other metadata), rather than having to read them. But even with reading them i am not sure about the concrete consequences of your proposal:
extractNativeEventFromFile(), when iterating over the envelope items, we drop everything that isn't an event; isn't that what you mean by "drop transactions"?sentry-java/sentry-android-core/src/main/java/io/sentry/android/core/NativeEventCollector.java
Lines 208 to 211 in b832e7c
NativeEventDatato just timestamp and File and re-read the data only once there is an actual match.The true cost is, of course, that we have to read, decode all envelopes, and deserialize every event, to verify that it is a native event and read its timestamp.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To be clear: We can make this part of the
NativeEventCollectorstream the envelopes (we only need item headers to skip anything but events) and the event (using aJsonObjectReaderon anInputStreamthat is bound by the payload size), avoiding loading the entire envelope and hydrating the full object graph of each event until we actually have a match with a tombstone (at which point we would just use the current implementationextractNativeEventFromFile()).I can't work on this today, but tomorrow you can have a basic implementation. That would eliminate the worry about loading a bunch of items into memory that we will never use.