Would static function composition be in the scope of this feature?

At the moment, higher-order functions in Dart are fairly limited since they require knowing the full prototype of the decorated function.

An example would be a debounce utility:

void Function() debouce(Duration duration, void Function() decorated) {
  Timer? timer;
  return () {
    timer?.cancel();
    timer = Timer(duration, () => decorated());
  };
}

which allows us to, instead of:

class Example {
  void doSomething() {

  }
}

write:

class Example {
  final doSomething = debounce(Duration(seconds: 1), () {
  
   });
}

but that comes with a few drawbacks:

• obvious readability decrease
• our debounce utility is not very reusable. It works only on void Function(), but we'd like it to work for all functions.

With static meta-programming, our debounce could inject code in the class at compilation, such that we could write:

class Example {
  @Debounce(Duration(seconds: 1))
  void doSomething() {
    print('doSomething');
  }
  
  @Debounce(Duration(seconds: 1))
  void doSomethingElse(int value, {String named}) {
    print('doSomethingElse $value named: $named');
  }
}

There is a delicate balance re: static function composition, but there are certainly many useful things that could be done with it. I think ultimately it is something we would like to support as long as we can make it obvious enough that this wrapping is happening.

The specific balance would be around user confusion - we have a guiding principle that we don't want to allow overwriting of code in order to ensure that programs keep their original meaning. There are a lot of useful things you could do by simply wrapping a function in some other function (some additional ones might include uniform exception handling, analytics reporting, argument validation, etc). Most of these things would not change the meaning really of the original function, but the code is being "changed" in some sense by being wrapped.

Ultimately my sense is this is something we should try to support though. I think the usefulness probably outweighs the potential for doing weird things.

I like Lisp approach (in my opinion, the utmost language when it comes to meta-programming). Instead of defining a @Debounce or something alike, we would define new syntax that would simply expand to a regular method at compile-time. I don't know, however, how much complex is to make something like this considering Dart syntax.

For something like debounce, a more aspect-like approach seems preferable. Say, if you could declaratively wrap a function body with some template code:

class Example {
  void doSomething() with debounce(Duration(seconds: 1)) {
    print('doSomething');
  }
  
  void doSomethingElse(int value, {String named}) with debounce(Duration(seconds: 1)) {
    print('doSomethingElse $value named: $named');
  }
}

template debounce<R>(Duration duration) on R Function {
  template final Stopwatch? sw;
  template late R result;
  if (sw != null && sw.elapsed < duration) {
    return result;
  } else {
    (sw ??= Stopwatch()..start()).reset();
    return result = super;
  }
}

This defines a "function template" (really, a kind of function mixin) which can be applied to other functions.
It cannot change the signature of the function, but it can access arguments (by forwarding them as with templateName(arg)), and it can do things before and after the original body is run.
The template variables are per-template instantiation variables (just as we could declare static variables inside normal functions).

(Maybe we just need AspectD for Dart.)

It cannot change the signature of the function, but it can access arguments

But an important part of function composition is also the ability to inject parameters and ask for more parameters.

For example, a good candidate is functional stateless-widgets, to add a key parameter to the prototype and inject a context parameter.
This means the user would define:

@statelessWidget
Widget example(BuildContext context, {required String name}) {
  return Text(name);
}

and the resulting prototype after composition would be:

Widget Function({Key? key, required String name})

where the final code would be:

class _Example extends StatelessWidget {
  Example({Key? key, required String name}): super(key: key);

  final String name;

  @override
  Widget build(BuildContext) => originalExampleFunction(context, name: name);
}

Widget example({Key? key, required String name}) {
  return _Example(key: key, name: name);
}

I definitely agree we don't want to allow for changing the signature of the function from what was written. I don't think that is prohibitive though as long as you are allowed to generate a new function/method next to the existing one with the signature you want. The original function might be private in that case.

I don't think that is prohibitive though as long as you are allowed to generate a new function/method next to the existing one with the signature you want

That's what functional_widget does, but the consequence is that the developer experience is pretty bad.

A major issue is that it breaks the "go to definition" functionality because instead of being redirected to their function, users are redirected to the generated code

It also causes a lot of confusion around naming. Because it's common to want to have control on whether the generated class/function is public or private, but the original function to always be private.

By modifying the prototype instead, this gives more control to users over the name of the generated functions.

Allowing the signature to be modified has a lot of disadvantages as well. I think its probably worse to see a function which is written to have a totally different signature than it actually has, than to be navigated to a generated function (which you can then follow through to the real one). You can potentially blackbox those functions in the debugger as well so it skips right to the real one if you are stepping through.

I suppose this will allow generating fromJson and toJson methods at compile time for Json serialization ?

@bouraine

I suppose this will allow generating fromJson and toJson methods at compile time for Json serialization ?

Yes.

@tatumizer This issue is just for the general problem of static metaprogramming. What you describe would be one possible solution to it, although we are trying to avoid exposing a full AST api because that can make it hard to evolve the language in the future. See https://github.com/dart-lang/language/blob/master/working/static%20metaprogramming/intro.md for an intro into the general design direction we are thinking of here which I think is not necessarily so far off from what you describe (although the mechanics are different).

Great intro & docs.

Hopefully we'll stay (far far) away from annotations to develop/work with static meta programming?!

so it looks like copyWith is seen as a crown jewel of the upcoming facility

The main reason we use this as an example is its well understood by many people, and it is also actually particularly demanding in terms of features to actually implement due to the public api itself needing to be generated :).

The language needs some mechanism of dealing with default values, which has been a showstopper in dart from day one.

Can you elaborate? Default values for parameters are getting some important upgrades in null safe dart (at least the major loophole of being able to override them accidentally by passing null explicitly is closed).

Can you elaborate? Default values for parameters are getting some important upgrades in null safe dart (at least the major loophole of being able to override them accidentally by passing null explicitly is closed).

I believe the issue is that we cannot easily differentiate between copyWith(name: null) and copyWith() where the former should assign null to name and the latter just do nothing

freezed supports this, but only because it relies on factory constructors and interface to hide the internals of copyWith (that is in fact a copyWith({Object? name = _internalDefault}))

I believe the issue is that we cannot easily differentiate between copyWith(name: null) and copyWith() where the former should assign null to name and the latter just do nothing

Right, this is what I was describing which null safety actually does fix at least partially. You can make the parameter non-nullable (with a default), and then null can no longer be passed at all. Wrapping functions are required to copy the default value, basically it forces you to explicitly handle this does cause some extra boilerplate but is safe.

For nullable parameters you still can't differentiate (at least in the function wrapping case, if they don't provide a default as well)

Metaprogramming is a broad topic. How to rationalize? We should start with what gives the best bang for buck (based on use cases).

Draft topics for meta programming 'output' code:

1. Methods
2. Classes (shell)
3. Class members
4. Types
5. Enums
6. Statements (?)
7. Mixins (?)
8. Generics (?)

Also on output code:

Be able to visualize in some way the code generated into your program, at development time
(https://github.com/dart-lang/language/blob/master/working/static%20metaprogramming/intro.md#usability)

Would be great if this could work without saving the file, a IDE-like syntax (hidden) code running continuously if syntax is valid. I refuse to use build_runner's watch

Metaprograming opens doors to many nice features
For the data class thing, this is something i miss from Kotlin.
When i used Java we used @Data / @Value from Lombok that was some sort of generator, i guess having something like this would be enough for the data class's

Other language that does a great job at implementing macros is Haxe you can use Haxe language to define macros

I guess there are many challenges to implement this.

can we extend classes with analyzer plugin?
can we use external and patch like patches in sdk libraries for extending classes?
plugins for CFE?

I'm not sure if I like the idea having this added to Dart because the beauty of Dart is its simplicity. The fact that it isn't as concise as other languages it in reality an advantage because it makes Dart code really easy to read and to reason about.
I fear meta programming will kill this. How will a goto-definition in an IDE work with it? How discoverable and maintainable is such code?

I'm not sure if I like the idea having this added to Dart because the beauty of Dart is its simplicity. The fact that it isn't as concise as other languages it in reality an advantage because it makes Dart code really easy to read and to reason about.
I fear meta programming will kill this. How will a goto-definition in an IDE work with it? How discoverable and maintainable is such code?

I agree with this.
I like the idea of meta programming as long as it doesn't remove how readable and maintainable a Dart code is.

@escamoteur Writing less code does not make it more complicated necessarily. It can, I agree, if someone does not fully understand the new syntax. But the trade-off is obvious: time & the number of lines saved vs the need for someone to learn a few capabilities.

Generated code is normal simple code. I just suggested real-time code generation instead of running the builder every time or watching it to save. That way you get real time goto. But if you are using notepad then of course you need to run a process.

I'm not sure if I like the idea having this added to Dart because the beauty of Dart is its simplicity. The fact that it isn't as concise as other languages it in reality an advantage because it makes Dart code really easy to read and to reason about.
I fear meta programming will kill this. How will a goto-definition in an IDE work with it? How discoverable and maintainable is such code?

Just to be 100% clear, we are intensely focused on these exact questions. We will not ship something which does not integrate well with all of our tools and workflows. You should be able to read code and understand it, go to definition, step through the code in the debugger, get good error messages, get clear and comprehensible stack traces, etc.

@escamoteur Writing less code does not make it more complicated necessarily. It can, I agree, if someone does not fully understand the new syntax. But the trade-off is obvious: time & the number of lines saved vs the need for someone to learn a few capabilities.

Generated code is normal simple code. I just suggested real-time code generation instead of running the builder every time or watching it to save. That way you get real time goto. But if you are using notepad then of course you need to run a process.

In my honest opinion: things must be obvious, not magical.
Every time I have to read or develop in PHP, JS or C with preprocessors etc... I just hate it.
Too many magical stuff that you just can't read or debug easily.
Dart is the opposite of that without being boring as hell as Java.
In fact, there was a time that some Dart packages used to implement the noSuchMethod to create magical methods. Gee, what a pain.
Meta programming could be the next Dart transformers if it takes the glittering magical road.

Just to be 100% clear, we are intensely focused on these exact questions. We will not ship something which does not integrate well with all of our tools and workflows. You should be able to read code and understand it, go to definition, step through the code in the debugger, get good error messages, get clear and comprehensible stack traces, etc.

^ this

I'm not sure if I like the idea having this added to Dart because the beauty of Dart is its simplicity.

But there is nothing beautiful about writing data classes or running complicated and and slow code-generation tools.

I'm hoping this can lead to more simplicity not less. Vast mounds of code will be removed from our visible classes. StatefulWidget can maybe just go away? (compiler can run the split macro before it builds?). Things can be auto-disposed. Seems like this could hit a lot of pain points, not just data classes and serialization..

Since dart currently offers code generation for similar jobs-to-be-done, I'd suggest evaluating potential concerns with that consideration:

• Metaprogramming is not simple/obvious - can it be made at least as simple/obvious as codegen through tooling?
• Metaprogramming will be abused - Is there a reason to think it will be abused more than codegen? (potentially, if it provides better ergonomics)

On the other hand, besides being an upgrade from codegen for developers, metaprogramming could provide healthier means for language evolution beyond getting data classes done. Quoting Bryan Cantrill:

Another advantage of macros: they are so flexible and powerful that they allow for effective experimentation. For example, the propagation operator that I love so much actually started life as a try! macro; that this macro was being used ubiquitously (and successfully) allowed a language-based solution to be considered. Languages can be (and have been!) ruined by too much experimentation happening in the language rather than in how it’s used; through its rich macros, it seems that Rust can enable the core of the language to remain smaller — and to make sure that when it expands, it is for the right reasons and in the right way.
http://dtrace.org/blogs/bmc/2018/09/18/falling-in-love-with-rust/

PS @jakemac53 the observable link leads to a private google doc.

this would be fantastic if it allowed, the longed-for serialization for JSON natively without the need for manual code generation or reflection in time of execution

Today practically all applications depend on serialization for JSON, a modern language like dart should already have a form of native serialization in the language, being obliged to use manual codegen or typing serialization manually is something very unpleasant

My approach on a macro mechanism. Basically tagging a certain scope with a macro annotation, that refers to one or multiple classes to 1:1 replace the code virtually... like a projection. It's very easy to understand and QOL can be extended by providing utility classes.

#ToStringMaker() // '#' indicates macro and will rewrite all code in next scope
class Person {
    String name;
    int age;
}

// [REWRITTEN CODE] => displayed readonly in IDE
// class Person {
//    String name;
//    int age;
//
//    toString() => 'Person(vorname:$name, age:$age)'
// }

class ToStringMaker extends Macro {

    // fields and constructor can optionally obtain parameters

    @override
    String generate(String code, MacroContext context) { // MacroContext provides access to other Dart files in project and other introspection features
        var writer = DartClassWriter(code); // DartClassWriter knows the structure of Dart code

        writer.add('String toString() => \'${writer.className}(${writer.fields.map(field => '${field.name}:\${field.name}').join(', ')})\'');

        return writer.code; // substitute code for referenced scope
    }

}

@muddassar-lab more likely early 2024, but implementation work is picking up now (it had been on hold for some time due to records/patterns/extension types etc, which took priority). The experiments so far are looking good, at least from my perspective, but there is also still the possibility we wouldn't ship the feature at all if it fails to hit certain criteria in terms of the development experience and/or usefulness.

I see that macros exist in the experiments list, and it looks like the augment keyword does stuff.

Honestly, whether or not we get macros itself anytime soon I think being able to play with augmentations would be super cool.

Yes, augmentations likely do work already to some extent. The implementations are still at a very early stage though so it isn't really ready for people to start trying it out, you will quickly hit unimplemented things and things that behave incorrectly.

@muddassar-lab more likely early 2024, but implementation work is picking up now (it had been on hold for some time due to records/patterns/extension types etc, which took priority). The experiments so far are looking good, at least from my perspective, but there is also still the possibility we wouldn't ship the feature at all if it fails to hit certain criteria in terms of the development experience and/or usefulness.

Considering how unwieldy and tedious the current build runner is, I doubt the criteria about dev experience and usefulness would be a problem.

I still wonder if we couldn't improve build_runner.

Macros will be better no-matter what. But maybe it's worth degrading experience of build_runner package authors, so that resulting packages are more efficient.

In regards to IDE tooling, will generated augmentation libraries be generated as on-disk files alongside the user's files, or elsewhere in the file system?

Also, assuming that generated code is easily visible to the developer, if there's a compile time error with the generated code due to developer misuse, will the augmentation code still be generated with an error or will the code not be generated with an error appearing at the macro annotation site? Will macro authors have control over this?

E.g., given the following code:

// Dto class that is missing `@JsonSerializable()`
class UserDto {
  const UserDto(this.name);
  final String name;
}

// Serializer class that consumes dtos that are expected
// to be serializable
@Mappable(UserDto)
class UserMapper { }

Would the below semantically-invalid code be generated or not?

augment class UserMapper {
  Map<String, dynamic> serialize(UserDto data) => 
      data.toJson(); // error from a missing `toJson()`method on `UserDto`
}

In regards to IDE tooling, will generated augmentation libraries be generated as on-disk files alongside the user's files, or elsewhere in the file system?

Macros don't create actual files. Their output is somewhere in memory. There will be special IDE considerations to show those files. But generally you shouldn't have to read them IMO, unless you're a macro author.

Would the below semantically-invalid code be generated or not?

It depends on the macro.

Macros have the ability to emit custom compilation errors. Depending on how @Mappable is implemented, it could show an error on the annotation instead of generating anything.

But if the macro forgot to handle all error cases, you should see an error in the virtual file, yes.
Although I'd report that case as a bug in the macro. Then, the package should be updated to better handle errors for this case.

Macros don't create actual files. Their output is somewhere in memory. There will be special IDE considerations to show those files. But generally you shouldn't have to read them IMO, unless you're a macro author.

In-memory representation does make sense for macros, though I completely disagree with users not needing to read the generated code. In the case of a generated fromJson method, I would expect a runtime error in debug mode to show the call site that the exception threw at, e.g. if a value is missing from a json map. This is the same UX as with on-disk package:build generated files today.

Macros have the ability to emit custom compilation errors.

I can't find any info on error handling, off the top of your head do you know the source for the macro error APIs?

But if the macro forgot to handle all error cases, you should see an error in the virtual file, yes. Although I'd report that case as a bug in the macro.

My concern is not regarding handling or not handling errors at all, its about creating a better UX for showing the user what the cause of a macro error is.

Let's say that UserDto is correctly annotated with @JsonSerializable(), but a new Email field is added like below:

class Email {...} // NOTE: no @JsonSerializable() annotation

@JsonSerializable()
class UserDto {
  const UserDto(this.name, this.email);
  final String name;
  final Email email; // invalid type due to missing `JsonSerializable` annotation on `Email`
}

// macro-generated augmentation
augment class UserDto {
  Map<String, dynamic> toJson() => {
      'name': name,
      'email': email.toJson(), // ERROR: `Email` type is not serializable
    };
}

In this particular case, IMO the best UX is that Phase 1 and 2 of the JsonSerializable macro should complete, despite the user's misuse of the macro. Another way to look at it is that the method declaration of toJson is valid (no namespace conflicts, etc.), but the implementation is invalid since email cannot be serialized - i.e. only Phase 3 fails.

This subtle difference in implementation makes the most difference for nested macro consumers. Since Phase 2 of UserDto.toJson generation was in fact completed, the UserMapper would not display compile time errors since the method declaration does exist:

@Mappable(UserDto)
class UserMapper { }

// macro-generated augmentation
augment class UserMapper {
  Map<String, dynamic> serialize(UserDto data) =>  data.toJson(); // NO ERROR
}

Otherwise, if the entire UserDto augmented library were to not generate, any macros dependent on the output of that macro will also fail, causing the user's project to light up like a red christmas tree with compile time errors for all dependent macros. Without a terminal based output like we have with build_runner, it will be harder for developers to locate the source of the error.

The better UX IMO is to show the user the single source of error (at email.toJson()), rather than show dozens of errors elsewhere.

However, other macro applications may require different UX or feel differently. What I'm suggesting is that package:_macros allows the generation either way (maybe as long as its syntactically valid?), and that the to-generate/not-to-generate decision be left up to the macro authors.

Macros don't create actual files. Their output is somewhere in memory. There will be special IDE considerations to show those files. But generally you shouldn't have to read them IMO, unless you're a macro author.

Ability to inspect generated code and insert breakpoints is important for macro users too.

(PS: I'm not arguing for generating files on disk. Special IDE support is fine.)

There will be some way of seeing generated code, I definitely think it will be important. If there is one thing package:build got right, I think it was this.

It will be tricky for some situations, in particular non-IDE use cases, like seeing a stack trace from production which points at generated code. We will have to work out all the details here but it is definitely possible to come up with solutions here.

I can't find any info on error handling, off the top of your head do you know the source for the macro error APIs?

The work in progress APIs are here https://github.com/dart-lang/sdk/tree/main/pkg/_fe_analyzer_shared/lib/src/macros/api, see specifically https://github.com/dart-lang/sdk/blob/main/pkg/_fe_analyzer_shared/lib/src/macros/api/builders.dart#L16. Any macro can also throw, and that will get converted into an error diagnostic attached to the macro annotation. But they can create diagnostics explicitly attached to specific nodes using this API which should be quite helpful.

However, other macro applications may require different UX or feel differently. What I'm suggesting is that package:_macros allows the generation either way (maybe as long as its syntactically valid?), and that the to-generate/not-to-generate decision be left up to the macro authors.

This is exactly how it will work. Macro authors can choose to only emit a diagnostic and no code, or intentionally emit broken code and rely on the compile time errors from that, or both. If a macro emits code, it will be visible, even if it also emits some error diagnostics.

I definitely agree that there can be a lot of pain if a macro fails in a way that causes a lot of other errors in the program. Mostly, I think this is something the analyzer team is going to need to discuss. The specification does not specifically say how this should be handled. For instance, the analyzer could choose to not show any errors related to macro classes which had a macro application that failed, and instead only show the macro failures. That wouldn't catch all cases, but likely would help. It is a fine line to draw though, and we will just need to explore what feels right when we are a bit further along imo.

Whether subsequent macro applications on the same declaration should even run after a single failed one is another open question.

@jakemac53 What happened to the idea of "const reflection" as you mentioned here ? Macros also sounds like we should have const functions somehow, since it's code executing at compilation phase. Allowing Something(const () => true)

Also is it possible to put a debugger during the compilation phase a when macro would be running ?

@jakemac53 @cedvdb something like zig comptime would be very flexible and powerful and would avoid using runtime reflection

https://ziglang.org/documentation/master/#comptime
https://kristoff.it/blog/what-is-zig-comptime/

@jakemac53 @cedvdb something like zig comptime would be very flexible and powerful and would avoid using runtime reflection

https://ziglang.org/documentation/master/#comptime https://kristoff.it/blog/what-is-zig-comptime/

doesn't Dart compiler already does that kind of thing when it encounters const stuff?

at least that is what happens with generated JS code from dart2js.

@jodinathan it seems like it allows const functions

@jakemac53 What happened to the idea of "const reflection" as you mentioned here ? Macros also sounds like we should have const functions somehow, since it's code executing at compilation phase. Allowing Something(const () => true)

Macros do run at compile time, but being more restricted than general const functions has some compile time advantages. We did explore enhanced const evaluation via the const evaluator (basically, this is just an interpreter). But it ends up being much more complicated (and slow) to compile programs and manage invalidation, in particular for large modular builds such as we have internally.

It also would be a lot of work for us to maintain an interpreter which is fully spec compliant. Macros are actually compiled and executed like regular dart programs, and don't need an interpreter. The boundaries of what code runs at compile time or not are also a lot more well defined, which also helps for compilation/invalidation.

Considering the upcoming macros feature I've created a Dart Element/Node Tree visualizer for VSCode: https://marketplace.visualstudio.com/items?itemName=jodirez.dart-element-tree-viewer.

It think It can be useful to devs to get to know the Element properties and maybe give us ideas for source generation stuff.

The extension is completely written in Dart.

Since this moved to "Being implemented" does it mean that static metaprogramming is sure to become a feature in dart, or is it still possible that the implementation will have problems and revert back?

The proposal has not yet moved to "accepted" in the language repo - that is the signal that it is for sure a go. We could still run into implementation or user experience issues in theory, that prevent us from shipping it. But, I am generally getting more confident and not less, the further along we get.

I would expect more tweaks to the proposal for a while, to work around smaller issues, or just because we think its a better design.

I'm curious if/how the following read/write scenarios (which are more-or-less supported by build_runner today) will be supported within the macros APIs:

• analyzing AST nodes of a given Declaration, e.g. function body, doc comments
• analyzing non-Dart code (e.g. a build.yaml file for configuration info about the project, or an OpenAPI json spec for generating a Dart REST API)
• producing non-Dart code (e.g. in a txt or json file)

As it stands, the macros APIs are focused solely on introspecting types and declarations, not implementation details (this obviously makes sense). On the other end of the spectrum, the Resources part of the proposal notes that macros will likely have read-only access to project files. So I assume that somewhere in between there lies an ability to access non-declaration related data.

So would the above use cases still be technically possible? To what extent would they be non-advisable, if at all?

analyzing AST nodes of a given Declaration, e.g. function body, doc comments

Definitely not in V1. Not sure about later, but almost certainly we would only expose that information for the immediately annotated declaration, or possibly the surrounding library, if anything.

analyzing non-Dart code (e.g. a build.yaml file for configuration info about the project, or an OpenAPI json spec for generating a Dart REST API)

The Resource API is the intended way to support reading non-Dart files. It still has to be driven by an annotation in Dart code somewhere. I could see a pattern where users create empty libraries with a macro only, which points at a resource that is some data file, and it generates the entire contents of the library from that.

producing non-Dart code (e.g. in a txt or json file)

Very unlikely.

It would be important for the macro API to support the possibility of reading project files, for example in AngularDart to read html files of a component's template and generate the ComponenteNgFactory