Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Defer image decoding when scrolling fast #49389

Merged
merged 14 commits into from Jan 30, 2020
5 changes: 5 additions & 0 deletions packages/flutter/lib/src/painting/image_cache.dart
Expand Up @@ -227,6 +227,11 @@ class ImageCache {
return result;
}

/// Returns whether this `key` has been previously added by [putIfAbsent].
bool containsKey(Object key) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a break.

return _pendingImages[key] != null || _cache[key] != null;
}

// Remove images from the cache until both the length and bytes are below
// maximum, or the cache is empty.
void _checkCacheSize() {
Expand Down
106 changes: 96 additions & 10 deletions packages/flutter/lib/src/painting/image_provider.dart
Expand Up @@ -182,6 +182,43 @@ typedef DecoderCallback = Future<ui.Codec> Function(Uint8List bytes, {int cacheW
///
/// The following image formats are supported: {@macro flutter.dart:ui.imageFormats}
///
/// ## Lifecycle of resolving an image
///
/// The [ImageProvider] goes through the following lifecycle to resolve an
/// image, once the [resolve] method is called:
///
/// 1. Create an [ImageStream] using [createStream] to return to the caller.
/// This stream will be used to communicate back to the caller when the
/// image is decoded and ready to display, or when an error occurs.
/// 2. Obtain the key for the image using [obtainKey].
/// Calling this method can throw exceptions into the zone asynchronously
/// or into the callstack synchronously. To handle that, an error handler
/// is created that catches both synchronous and asynchronous errors, to
/// make sure errors can be routed to the correct consumers.
/// The error handler is passed on to [resolveStreamForKey] and the
/// [ImageCache].
/// 3. If the key is successfully obtained, schedule resolution of the image
/// using that key. This is handled by [resolveStreamForKey]. That method
/// may fizzle if it determines the image is no longer necessary, use the
/// provided [ImageErrorListener] to report an error, set the completer
/// from the cache if possible, or call [load] to fetch the encoded image
/// bytes and schedule decoding.
/// 4. The [load] method is responsible for both fetching the encoded bytes
/// and decoding them using the provided [DecoderCallback]. It is called
/// in a context that uses the [ImageErrorListener] to report errors back.
///
/// Subclasses normally only have to implement the [load] and [obtainKey]
/// methods. A subclass that needs finer grained control over the [ImageStream]
/// type must override [createStream]. A subclass that needs finer grained
/// control over the resolution, such as delaying calling [load], must override
/// [resolveStreamForKey].
///
/// The [resolve] method is marked as [nonVirtual] so that [ImageProvider]s can
/// be properly composed, and so that the base class can properly set up error
/// handling for subsequent methods.
///
/// ## Using an [ImageProvider]
///
/// {@tool snippet}
///
/// The following shows the code required to write a widget that fully conforms
Expand Down Expand Up @@ -270,10 +307,34 @@ abstract class ImageProvider<T> {
/// This is the public entry-point of the [ImageProvider] class hierarchy.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add a section to the class docs (## foo) that discusses the lifecycle of this class and the order in which the various methods are called, by whom, when, and so on. The class is complicated enough now that an overview is necessary.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

///
/// Subclasses should implement [obtainKey] and [load], which are used by this
/// method.
/// method. If they need to change the implementation of [ImageStream] used,
/// they should override [createStream]. If they need to manage the actual
/// resolution of the image, they should override [resolveStreamForKey].
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a link to the lifecycle explanation of this method in the class comment?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done - I think.

///
/// See the Lifecycle documentation on [ImageProvider] for more information.
@nonVirtual
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a break.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why must it be nonVirtual? I mean, I get that it's useful for catching people who are using the API the old way, but is it strictly necessary? What if someone wants to take an entirely different approach?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because if some other provider overrides it, its new logic will never get called by the new kind of provider I'm making that would wrap it.

I imagine other providers would have this problem. Plus the logic in it is very complicated.

Alternatively, we could update docs to say "if you override this, composed providers may not work and you should consider overriding useKey instead" - but I'm inclined to think people are more likely to see an analyzer warning than a doc comment.

ImageStream resolve(ImageConfiguration configuration) {
assert(configuration != null);
final ImageStream stream = ImageStream();
final ImageStream stream = createStream(configuration);
// Load the key (potentially asynchronously), set up an error handling zone,
// and call resolveStreamForKey.
_createErrorHandlerAndKey(configuration, stream);
return stream;
}

/// Called by [resolve] to create the [ImageStream] it returns.
///
/// Subclasses should override this instead of [resolve] if they need to
/// return some subclass of [ImageStream]. The stream created here will be
/// passed to [resolveStreamForKey].
@protected
ImageStream createStream(ImageConfiguration configuration) {
return ImageStream();
}

void _createErrorHandlerAndKey(ImageConfiguration configuration, ImageStream stream) {
assert(configuration != null);
assert(stream != null);
T obtainedKey;
bool didError = false;
Future<void> handleError(dynamic exception, StackTrace stack) async {
Expand Down Expand Up @@ -322,17 +383,42 @@ abstract class ImageProvider<T> {
}
key.then<void>((T key) {
obtainedKey = key;
final ImageStreamCompleter completer = PaintingBinding.instance.imageCache.putIfAbsent(
key,
() => load(key, PaintingBinding.instance.instantiateImageCodec),
onError: handleError,
);
if (completer != null) {
stream.setCompleter(completer);
try {
resolveStreamForKey(configuration, stream, key, handleError);
} catch (error, stackTrace) {
handleError(error, stackTrace);
}
}).catchError(handleError);
});
return stream;
}

/// Called by [resolve] with the key returned by [obtainKey].
///
/// Subclasses should override this method rather than calling [obtainKey] if
/// they need to use a key directly. The [resolve] method installs appropriate
/// error handling guards so that errors will bubble up to the right places in
/// the framework, and passes those guards along to this method via the
/// [handleError] parameter.
///
/// It is safe for the implementation of this method to call [handleError]
/// multiple times if multiple errors occur, or if an error is thrown both
/// synchronously into the current part of the stack and thrown into the
/// enclosing [Zone].
///
/// The default implementation uses the key to interact with the [ImageCache],
/// calling [ImageCache.putIfAbsent] and notifying listeners of the [stream].
/// Implementers that do not call super are expected to correctly use the
/// [ImageCache].
@protected
void resolveStreamForKey(ImageConfiguration configuration, ImageStream stream, T key, ImageErrorListener handleError) {
final ImageStreamCompleter completer = PaintingBinding.instance.imageCache.putIfAbsent(
key,
() => load(key, PaintingBinding.instance.instantiateImageCodec),
onError: handleError,
);
if (completer != null) {
stream.setCompleter(completer);
}
}

/// Evicts an entry from the image cache.
Expand Down
1 change: 1 addition & 0 deletions packages/flutter/lib/src/painting/image_stream.dart
Expand Up @@ -340,6 +340,7 @@ abstract class ImageStreamCompleter extends Diagnosticable {
/// is false after calling `super.removeListener()`, and if so, stopping that
/// same work.
@protected
@visibleForTesting
bool get hasListeners => _listeners.isNotEmpty;

/// Adds a listener callback that is called whenever a new concrete [ImageInfo]
Expand Down
72 changes: 72 additions & 0 deletions packages/flutter/lib/src/widgets/disposable_build_context.dart
@@ -0,0 +1,72 @@
// Copyright 2014 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

import 'framework.dart';

/// Provides non-leaking access to a [BuildContext].
///
/// A [BuildContext] is only valid if it is pointing to an active [Element].
/// Once the [Element.dispose] method is called, the [BuildContext] should not
/// be accessed further. This class makes it possible for a [StatefulWidget] to
/// share its build context safely with other objects.
///
/// Creators of this object must guarantee the following:
///
/// 1. They create this object at or after [State.initState] but before
/// [State.dispose]. In particular, do not attempt to create this from the
/// constructor of a state.
/// 2. They call [dispose] from [State.dispose].
///
/// This object will not hold on to the [State] after disposal.
@optionalTypeArgs
class DisposableBuildContext<T extends State> {
/// Creates an object that provides access to a [BuildContext] without leaking
/// a [State].
///
/// Creators must call [dispose] when the [State] is disposed.
///
/// The [State] must not be null, and [State.mounted] must be true.
DisposableBuildContext(this._state)
: assert(_state != null),
assert(_state.mounted, 'A DisposableBuildContext was given a BuildContext for an Element that is not mounted.');

T _state;

/// Provides safe access to the build context.
///
/// If [dispose] has been called, will return null.
///
/// Otherwise, asserts the [_state] is still mounted and returns its context.
BuildContext get context {
assert(_debugValidate());
if (_state == null) {
return null;
}
return _state.context;
}

/// Called from asserts or tests to determine whether this object is in a
/// valid state.
///
/// Always returns true, but will assert if [dispose] has not been called
/// but the state this is tracking is unmounted.
bool _debugValidate() {
assert(
_state == null || _state.mounted,
'A DisposableBuildContext tried to access the BuildContext of a disposed '
'State object. This can happen when the creator of this '
'DisposableBuildContext fails to call dispose when it is disposed.',
);
return true;
}


/// Marks the [BuildContext] as disposed.
///
/// Creators of this object must call [dispose] when their [Element] is
/// unmounted, i.e. when [State.dispose] is called.
void dispose() {
_state = null;
}
}
11 changes: 10 additions & 1 deletion packages/flutter/lib/src/widgets/image.dart
Expand Up @@ -13,9 +13,11 @@ import 'package:flutter/semantics.dart';

import 'basic.dart';
import 'binding.dart';
import 'disposable_build_context.dart';
import 'framework.dart';
import 'localizations.dart';
import 'media_query.dart';
import 'scroll_aware_image_provider.dart';
import 'ticker_provider.dart';

export 'package:flutter/painting.dart' show
Expand Down Expand Up @@ -946,18 +948,21 @@ class _ImageState extends State<Image> with WidgetsBindingObserver {
bool _invertColors;
int _frameNumber;
bool _wasSynchronouslyLoaded;
DisposableBuildContext<State<Image>> _scrollAwareContext;

@override
void initState() {
super.initState();
WidgetsBinding.instance.addObserver(this);
_scrollAwareContext = DisposableBuildContext<State<Image>>(this);
}

@override
void dispose() {
assert(_imageStream != null);
WidgetsBinding.instance.removeObserver(this);
_stopListeningToStream();
_scrollAwareContext.dispose();
super.dispose();
}

Expand Down Expand Up @@ -1006,8 +1011,12 @@ class _ImageState extends State<Image> with WidgetsBindingObserver {
}

void _resolveImage() {
final ScrollAwareImageProvider provider = ScrollAwareImageProvider<dynamic>(
context: _scrollAwareContext,
imageProvider: widget.image,
);
final ImageStream newStream =
widget.image.resolve(createLocalImageConfiguration(
provider.resolve(createLocalImageConfiguration(
context,
size: widget.width != null && widget.height != null ? Size(widget.width, widget.height) : null,
));
Expand Down
109 changes: 109 additions & 0 deletions packages/flutter/lib/src/widgets/scroll_aware_image_provider.dart
@@ -0,0 +1,109 @@
// Copyright 2014 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

import 'dart:async';

import 'package:flutter/painting.dart';
import 'package:flutter/scheduler.dart';

import 'disposable_build_context.dart';
import 'framework.dart';
import 'scrollable.dart';

/// An [ImageProvider] that makes use of
/// [Scollable.recommendDeferredLoadingForContext] to avoid loading images when
/// rapidly scrolling.
///
/// This provider assumes that its wrapped [imageProvider] correctly uses the
/// [ImageCache], and does not attempt to re-acquire or decode images in the
/// cache.
///
/// Calling [resolve] on this provider will cause it to obtain the image key
goderbauer marked this conversation as resolved.
Show resolved Hide resolved
/// and then check the following:
///
/// 1. If the returned [ImageStream] has been completed, end. This can happen
/// if the caller sets the completer on the stream.
/// 2. If the [ImageCache] has a completer for the key for this image, ask the
/// wrapped provider to resolve.
/// This can happen if the image was precached, or another [ImageProvider]
/// already resolved the same image.
/// 3. If the [context] has been disposed, end. This can happen if the caller
/// has been disposed and is no longer interested in resolving the image.
/// 4. If the widget is scrolling with high velocity at this point in time,
/// wait until the beginning of the next frame and go back to step 1.
/// 5. Delegate loading the image to the wrapped provider and finish.
///
/// If the cycle ends at steps 1 or 3, the [ImageStream] will never be marked as
/// complete and listeners will not be notified.
///
/// The [Image] widget wraps its incoming providers with this provider to avoid
/// overutilization of resources for images that would never appear on screen or
/// only be visible for a very brief period.
@optionalTypeArgs
class ScrollAwareImageProvider<T> extends ImageProvider<T> {
/// Creates a [ScrollingAwareImageProvider].
///
/// The [context] object is the [BuildContext] of the [State] using this
/// provider. It is used to determine scrolling velocity during [resolve]. It
/// must not be null.
///
/// The [imageProvider] is used to create a key and load the image. It must
/// not be null, and is assumed to interact with the cache in the normal way
/// that [ImageProvider.resolveStreamForKey] does.
const ScrollAwareImageProvider({
@required this.context,
@required this.imageProvider,
}) : assert(context != null),
assert(imageProvider != null);

/// The context that may or may not be enclosed by a [Scrollable].
///
/// Once [DisposableBuildContext.dispose] is called on this context,
/// the provider will stop trying to resolve the image if it has not already
/// been resolved.
final DisposableBuildContext context;

/// The wrapped image provider to delegate [obtainKey] and [load] to.
final ImageProvider<T> imageProvider;

@override
void resolveStreamForKey(
ImageConfiguration configuration,
ImageStream stream,
T key,
ImageErrorListener handleError,
) {
// Something managed to complete the stream. Nothing left to do.
if (stream.completer != null) {
return;
}
// Something else got this image into the cache. Return it.
if (PaintingBinding.instance.imageCache.containsKey(key)) {
imageProvider.resolveStreamForKey(configuration, stream, key, handleError);
}
// The context has gone out of the tree - ignore it.
if (context.context == null) {
return;
}
// Something still wants this image, but check if the context is scrolling
// too fast before scheduling work that might never show on screen.
// Try to get to end of the frame callbacks of the next frame, and then
// check again.
if (Scrollable.recommendDeferredLoadingForContext(context.context)) {
SchedulerBinding.instance.scheduleFrameCallback((_) {
scheduleMicrotask(() => resolveStreamForKey(configuration, stream, key, handleError));
});
return;
}
// We are in the tree, we're not scrolling too fast, the cache doens't
// have our image, and no one has otherwise completed the stream. Go.
imageProvider.resolveStreamForKey(configuration, stream, key, handleError);
}

@override
ImageStreamCompleter load(T key, DecoderCallback decode) => imageProvider.load(key, decode);

@override
Future<T> obtainKey(ImageConfiguration configuration) => imageProvider.obtainKey(configuration);
}
2 changes: 2 additions & 0 deletions packages/flutter/lib/widgets.dart
Expand Up @@ -31,6 +31,7 @@ export 'src/widgets/color_filter.dart';
export 'src/widgets/container.dart';
export 'src/widgets/debug.dart';
export 'src/widgets/dismissible.dart';
export 'src/widgets/disposable_build_context.dart';
export 'src/widgets/drag_target.dart';
export 'src/widgets/draggable_scrollable_sheet.dart';
export 'src/widgets/editable_text.dart';
Expand Down Expand Up @@ -78,6 +79,7 @@ export 'src/widgets/raw_keyboard_listener.dart';
export 'src/widgets/routes.dart';
export 'src/widgets/safe_area.dart';
export 'src/widgets/scroll_activity.dart';
export 'src/widgets/scroll_aware_image_provider.dart';
export 'src/widgets/scroll_configuration.dart';
export 'src/widgets/scroll_context.dart';
export 'src/widgets/scroll_controller.dart';
Expand Down