[CIR][CIRGen][Lowering] Add support for attribute annotate (llvm#804)

The main purpose of this PR is to add support for C/C++ attribute
annotate. The PR involves both CIR generation and Lowering Prepare. In
the rest of this description, we first introduce the concept of
attribute annotate, then talk about expectations of LLVM regarding
annotation, after it, we describe how ClangIR handles it in this PR.
Finally, we list trivial differences between LLVM code generated by
clang codegen and ClangIR codegen.
**The concept of attribute annotate. and expected LLVM IR**
the following is C code example of annotation.
say in example.c
`int *b __attribute__((annotate("withargs", "21", 12 )));
int *a __attribute__((annotate("oneargs", "21", )));
int *c __attribute__((annotate("noargs")));
`
here "withargs" is the annotation string, "21" and 12 are arguments for
this annotation named "withargs". LLVM-based compiler is expected keep
these information and build a global variable capturing all annotations
used in the translation unit when emitting into LLVM IR. This global
variable itself is **not** constant, but will be initialized with
constants that are related to annotation representation, e.g. "withargs"
should be literal string variable in IR.

This global variable has a fixed name "llvm.global.annotations", and its
of array of struct type, and should be initialized with a const array of
const structs, each const struct is a representation of an annotation
site, which has 5-field.
[ptr to global var/func annotated, ptr to translation unit string const,
line_no, annotation_name, ptr to arguments const]
annotation name string and args constants, as well as this global var
should be in section "llvm.metadata".
e.g. In the above example, 
We shall have following in the generated LLVM IR like the following 
```
@b = global ptr null, align 8
@.str = private unnamed_addr constant [9 x i8] c"withargs\00", section "llvm.metadata"
@.str.1 = private unnamed_addr constant [10 x i8] c"example.c\00", section "llvm.metadata"
@.str.2 = private unnamed_addr constant [3 x i8] c"21\00", align 1
@.args = private unnamed_addr constant { ptr, i32 } { ptr @.str.2, i32 12 }, section "llvm.metadata"
@A = global ptr null, align 8
@.str.3 = private unnamed_addr constant [8 x i8] c"oneargs\00", section "llvm.metadata"
@.args.4 = private unnamed_addr constant { ptr } { ptr @.str.2 }, section "llvm.metadata"
@c = global ptr null, align 8
@.str.5 = private unnamed_addr constant [7 x i8] c"noargs\00", section "llvm.metadata"
@llvm.global.annotations = appending global [3 x { ptr, ptr, ptr, i32, ptr }] [{ ptr, ptr, ptr, i32, ptr } { ptr @b, ptr @.str, ptr @.str.1, i32 1, ptr @.args }, { ptr, ptr, ptr, i32, ptr } { ptr @A, ptr @.str.3, ptr @.str.1, i32 2, ptr @.args.4 }, { ptr, ptr, ptr, i32, ptr } { ptr @c, ptr @.str.5, ptr @.str.1, i32 3, ptr null }], section "llvm.metadata"
```
notice that since variable c's annotation has no arg, the last field of
its corresponding annotation entry is a nullptr.

**ClangIR's handling of annotations**
In CIR, we introduce AnnotationAttr to GlobalOp and FuncOp to record its
annotations. That way, we are able to make fast query about annotation
if in future a CIR pass is interested in them.
We leave the work of generating const variables as well as global
annotations' var to LLVM lowering. But at LoweringPrepare we collect all
annotations and create a module attribute "cir.global_annotations" so to
facilitate LLVM lowering.

**Some implementation details and trivial differences between clangir
generated LLVM code and vanilla LLVM code**
1. I suffix names of constants generated for annotation purpose with
".annotation" to avoid redefinition, but clang codegen doesn't do it.
3. clang codegen seems to visit FuncDecls in slightly different orders
than CIR, thus, sometimes the order of elements of the initial value
const array for llvm.global.annotations var is different from clang
generated LLVMIR, it should be trivial, as I don't expect consumer of
this var is assuming a fixed order of collecting annotations.

Otherwise, clang codegen and clangir pretty much generate same LLVM IR
for annotations!

clang/include/clang/CIR/Dialect/IR/CIRAttrs.td

@@ -1101,6 +1101,77 @@ def BitfieldInfoAttr : CIR_Attr<"BitfieldInfo", "bitfield_info"> {
   ];
 }
 
+//===----------------------------------------------------------------------===//
+// AnnotationAttr
+//===----------------------------------------------------------------------===//
+
+def AnnotationAttr : CIR_Attr<"Annotation", "annotation"> {
+  let summary = "Annotation attribute for global variables and functions";
+  let description = [{
+    Represent C/C++ attribute of annotate in CIR.
+    Example C code:
+    ```
+     int *a __attribute__((annotate("testptr", "21", 12 )));
+    ```
+    In this example code, the `AnnotationAttr` has annotation name "testptr",
+    and arguments "21" and 12 constitutes an `ArrayAttr` type parameter `args`
+    for global variable `a`.
+    In CIR, the attribute for above annotation looks like:
+    ```
+    [#cir.annotation<name = "withargs", args = ["21", 12 : i32]>]
+    ```
+  }];
+
+  // The parameter args is empty when there is no arg.
+  let parameters = (ins "StringAttr":$name,
+                        "ArrayAttr":$args);
+
+  let assemblyFormat = "`<` struct($name, $args) `>`";                  
+
+  let extraClassDeclaration = [{
+    bool isNoArgs() const { return getArgs().empty(); };
+  }];
+}
+
+//===----------------------------------------------------------------------===//
+// GlobalAnnotationValuesAttr
+//===----------------------------------------------------------------------===//
+
+def GlobalAnnotationValuesAttr : CIR_Attr<"GlobalAnnotationValues",
+                                          "global_annotations"> {
+  let summary = "Array of annotations, each element consists of name of"
+                "a global var or func and one of its annotations";
+  let description = [{
+    This is annotation value array, which holds the annotation
+    values for all global variables and functions in a module.
+    This array is used to create the initial value of a global annotation
+    metadata variable in LLVM IR.
+    Example C code:
+    ```
+    double *a __attribute__((annotate("withargs", "21", 12 )));
+    int *b __attribute__((annotate("withargs", "21", 12 )));
+    void *c __attribute__((annotate("noargvar")));
+    void foo(int i) __attribute__((annotate("noargfunc"))) {}
+    ```
+    After CIR lowering prepare pass, compiler generates a 
+    `GlobalAnnotationValuesAttr` like the following:
+    ```
+    #cir<global_annotations [
+      ["a", #cir.annotation<name = "withargs", args = ["21", 12 : i32]>],
+      ["b", #cir.annotation<name = "withargs", args = ["21", 12 : i32]>],
+      ["c", #cir.annotation<name = "noargvar", args = []>],
+      ["foo", #cir.annotation<name = "noargfunc", args = []>]]>
+    ```
+  }];
+
+  let parameters = (ins "ArrayAttr":$annotations);
+
+  let assemblyFormat = [{ $annotations }];
+
+  // Enable verifier.
+  let genVerifyDecl = 1;
+}
+
 include "clang/CIR/Dialect/IR/CIROpenCLAttrs.td"
 
 #endif // MLIR_CIR_DIALECT_CIR_ATTRS

clang/include/clang/CIR/Dialect/IR/CIROps.td

@@ -2205,7 +2205,7 @@ def TLSModel : I32EnumAttr<
     [TLS_GeneralDynamic, TLS_LocalDynamic, TLS_InitialExec, TLS_LocalExec]> {
   let cppNamespace = "::mlir::cir";
 }
-  
+
 def GlobalOp : CIR_Op<"global",
                       [DeclareOpInterfaceMethods<RegionBranchOpInterface>,
                        DeclareOpInterfaceMethods<CIRGlobalValueInterface>,
@@ -2255,7 +2255,8 @@ def GlobalOp : CIR_Op<"global",
                        UnitAttr:$dsolocal,
                        OptionalAttr<I64Attr>:$alignment,
                        OptionalAttr<ASTVarDeclInterface>:$ast,
-                       OptionalAttr<StrAttr>:$section);
+                       OptionalAttr<StrAttr>:$section,
+                       OptionalAttr<ArrayAttr>:$annotations);
   let regions = (region AnyRegion:$ctorRegion, AnyRegion:$dtorRegion);
   let assemblyFormat = [{
        ($sym_visibility^)?
@@ -2268,6 +2269,7 @@ def GlobalOp : CIR_Op<"global",
        (`addrspace` `(` custom<GlobalOpAddrSpace>($addr_space)^ `)`)?
        $sym_name
        custom<GlobalOpTypeAndInitialValue>($sym_type, $initial_value, $ctorRegion, $dtorRegion)
+       ($annotations^)?
        attr-dict
   }];
 
@@ -3100,6 +3102,7 @@ def FuncOp : CIR_Op<"func", [
                        OptionalAttr<FlatSymbolRefAttr>:$aliasee,
                        OptionalAttr<GlobalCtorAttr>:$global_ctor,
                        OptionalAttr<GlobalDtorAttr>:$global_dtor,
+                       OptionalAttr<ArrayAttr>:$annotations,
                        OptionalAttr<AnyASTFunctionDeclAttr>:$ast);
   let regions = (region AnyRegion:$body);
   let skipDefaultBuilders = 1;

clang/lib/CIR/CodeGen/CIRGenModule.cpp

@@ -484,6 +484,13 @@ void CIRGenModule::buildGlobal(GlobalDecl GD) {
 
   // Ignore declarations, they will be emitted on their first use.
   if (const auto *FD = dyn_cast<FunctionDecl>(Global)) {
+    // Update deferred annotations with the latest declaration if the function
+    // was already used or defined.
+    if (FD->hasAttr<AnnotateAttr>()) {
+      StringRef MangledName = getMangledName(GD);
+      if (getGlobalValue(MangledName))
+        deferredAnnotations[MangledName] = FD;
+    }
     // Forward declarations are emitted lazily on first use.
     if (!FD->doesThisDeclarationHaveABody()) {
       if (!FD->doesDeclarationForceExternallyVisibleDefinition())
@@ -595,7 +602,8 @@ void CIRGenModule::buildGlobalFunctionDefinition(GlobalDecl GD,
   if (const DestructorAttr *DA = D->getAttr<DestructorAttr>())
     AddGlobalDtor(Fn, DA->getPriority(), true);
 
-  assert(!D->getAttr<AnnotateAttr>() && "NYI");
+  if (D->getAttr<AnnotateAttr>())
+    deferredAnnotations[getMangledName(GD)] = cast<ValueDecl>(D);
 }
 
 /// Track functions to be called before main() runs.
@@ -1232,7 +1240,7 @@ void CIRGenModule::buildGlobalVarDefinition(const clang::VarDecl *D,
   maybeHandleStaticInExternC(D, GV);
 
   if (D->hasAttr<AnnotateAttr>())
-    assert(0 && "not implemented");
+    addGlobalAnnotations(D, GV);
 
   // Set CIR's linkage type as appropriate.
   mlir::cir::GlobalLinkageKind Linkage =
@@ -2834,7 +2842,7 @@ void CIRGenModule::Release() {
   // TODO: PGOReader
   // TODO: buildCtorList(GlobalCtors);
   // TODO: builtCtorList(GlobalDtors);
-  // TODO: buildGlobalAnnotations();
+  buildGlobalAnnotations();
   // TODO: buildDeferredUnusedCoverageMappings();
   // TODO: CIRGenPGO
   // TODO: CoverageMapping
@@ -3188,3 +3196,68 @@ LangAS CIRGenModule::getGlobalVarAddressSpace(const VarDecl *D) {
 
   return getTargetCIRGenInfo().getGlobalVarAddressSpace(*this, D);
 }
+
+mlir::ArrayAttr CIRGenModule::buildAnnotationArgs(AnnotateAttr *attr) {
+  ArrayRef<Expr *> exprs = {attr->args_begin(), attr->args_size()};
+  if (exprs.empty()) {
+    return mlir::ArrayAttr::get(builder.getContext(), {});
+  }
+  llvm::FoldingSetNodeID id;
+  for (Expr *e : exprs) {
+    id.Add(cast<clang::ConstantExpr>(e)->getAPValueResult());
+  }
+  mlir::ArrayAttr &lookup = annotationArgs[id.ComputeHash()];
+  if (lookup)
+    return lookup;
+
+  llvm::SmallVector<mlir::Attribute, 4> args;
+  args.reserve(exprs.size());
+  for (Expr *e : exprs) {
+    if (auto *const strE =
+            ::clang::dyn_cast<clang::StringLiteral>(e->IgnoreParenCasts())) {
+      // Add trailing null character as StringLiteral->getString() does not
+      args.push_back(builder.getStringAttr(strE->getString()));
+    } else if (auto *const intE = ::clang::dyn_cast<clang::IntegerLiteral>(
+                   e->IgnoreParenCasts())) {
+      args.push_back(mlir::IntegerAttr::get(
+          mlir::IntegerType::get(builder.getContext(),
+                                 intE->getValue().getBitWidth()),
+          intE->getValue()));
+    } else {
+      llvm_unreachable("NYI");
+    }
+  }
+
+  lookup = builder.getArrayAttr(args);
+  return lookup;
+}
+
+mlir::cir::AnnotationAttr
+CIRGenModule::buildAnnotateAttr(clang::AnnotateAttr *aa) {
+  mlir::StringAttr annoGV = builder.getStringAttr(aa->getAnnotation());
+  mlir::ArrayAttr args = buildAnnotationArgs(aa);
+  return mlir::cir::AnnotationAttr::get(builder.getContext(), annoGV, args);
+}
+
+void CIRGenModule::addGlobalAnnotations(const ValueDecl *d,
+                                        mlir::Operation *gv) {
+  assert(d->hasAttr<AnnotateAttr>() && "no annotate attribute");
+  assert((isa<GlobalOp>(gv) || isa<FuncOp>(gv)) &&
+         "annotation only on globals");
+  llvm::SmallVector<mlir::Attribute, 4> annotations;
+  for (auto *i : d->specific_attrs<AnnotateAttr>())
+    annotations.push_back(buildAnnotateAttr(i));
+  if (auto global = dyn_cast<mlir::cir::GlobalOp>(gv))
+    global.setAnnotationsAttr(builder.getArrayAttr(annotations));
+  else if (auto func = dyn_cast<mlir::cir::FuncOp>(gv))
+    func.setAnnotationsAttr(builder.getArrayAttr(annotations));
+}
+
+void CIRGenModule::buildGlobalAnnotations() {
+  for (const auto &[mangledName, vd] : deferredAnnotations) {
+    mlir::Operation *gv = getGlobalValue(mangledName);
+    if (gv)
+      addGlobalAnnotations(vd, gv);
+  }
+  deferredAnnotations.clear();
+}

clang/lib/CIR/CodeGen/CIRGenModule.h

@@ -127,6 +127,22 @@ class CIRGenModule : public CIRGenTypeCache {
   /// for the same decl.
   llvm::DenseSet<clang::GlobalDecl> DiagnosedConflictingDefinitions;
 
+  /// -------
+  /// Annotations
+  /// -------
+
+  /// We do not store global annotations in the module here, instead, we store
+  /// each annotation as attribute of GlobalOp and FuncOp.
+  /// We defer creation of global annotation variable to LoweringPrepare
+  /// as CIR passes do not need to have a global view of all annotations.
+
+  /// Used for uniquing of annotation arguments.
+  llvm::DenseMap<unsigned, mlir::ArrayAttr> annotationArgs;
+
+  /// Store deferred function annotations so they can be emitted at the end with
+  /// most up to date ValueDecl that will have all the inherited annotations.
+  llvm::DenseMap<StringRef, const ValueDecl *> deferredAnnotations;
+
 public:
   mlir::ModuleOp getModule() const { return theModule; }
   CIRGenBuilderTy &getBuilder() { return builder; }
@@ -761,6 +777,24 @@ class CIRGenModule : public CIRGenTypeCache {
   void setNonAliasAttributes(GlobalDecl GD, mlir::Operation *GV);
   /// Map source language used to a CIR attribute.
   mlir::cir::SourceLanguage getCIRSourceLanguage();
+
+  /// Emit all the global annotations.
+  /// This actually only emits annotations for deffered declarations of
+  /// functions, because global variables need no deffred emission.
+  void buildGlobalAnnotations();
+
+  /// Emit additional args of the annotation.
+  mlir::ArrayAttr buildAnnotationArgs(clang::AnnotateAttr *attr);
+
+  /// Create cir::AnnotationAttr which contains the annotation
+  /// information for a given GlobalValue. Notice that a GlobalValue could
+  /// have multiple annotations, and this function creates attribute for
+  /// one of them.
+  mlir::cir::AnnotationAttr buildAnnotateAttr(clang::AnnotateAttr *aa);
+
+  /// Add global annotations for a global value.
+  /// Those annotations are emitted during lowering to the LLVM code.
+  void addGlobalAnnotations(const ValueDecl *d, mlir::Operation *gv);
 };
 } // namespace cir
 

clang/lib/CIR/Dialect/IR/CIRAttrs.cpp

@@ -545,6 +545,46 @@ void MethodAttr::print(AsmPrinter &printer) const {
   printer << '>';
 }
 
+//===----------------------------------------------------------------------===//
+// GlobalAnnotationValuesAttr definitions
+//===----------------------------------------------------------------------===//
+
+LogicalResult GlobalAnnotationValuesAttr::verify(
+    function_ref<::mlir::InFlightDiagnostic()> emitError,
+    mlir::ArrayAttr annotations) {
+  if (annotations.empty()) {
+    emitError()
+        << "GlobalAnnotationValuesAttr should at least have one annotation";
+    return failure();
+  }
+  for (auto &entry : annotations) {
+    auto annoEntry = ::mlir::dyn_cast<mlir::ArrayAttr>(entry);
+    if (!annoEntry) {
+      emitError() << "Element of GlobalAnnotationValuesAttr annotations array"
+                     " must be an array";
+      return failure();
+    } else if (annoEntry.size() != 2) {
+      emitError() << "Element of GlobalAnnotationValuesAttr annotations array"
+                  << " must be a 2-element array and you have "
+                  << annoEntry.size();
+      return failure();
+    } else if (!::mlir::isa<mlir::StringAttr>(annoEntry[0])) {
+      emitError() << "Element of GlobalAnnotationValuesAttr annotations"
+                     "array must start with a string, which is the name of "
+                     "global op or func it annotates";
+      return failure();
+    }
+    auto annoPart = ::mlir::dyn_cast<mlir::cir::AnnotationAttr>(annoEntry[1]);
+    if (!annoPart) {
+      emitError() << "The second element of GlobalAnnotationValuesAttr"
+                     "annotations array element must be of "
+                     "type AnnotationValueAttr";
+      return failure();
+    }
+  }
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // DynamicCastInfoAtttr definitions
 //===----------------------------------------------------------------------===//

clang/lib/CIR/Dialect/IR/CIRDialect.cpp

@@ -717,7 +717,7 @@ bool isIntOrBoolCast(mlir::cir::CastOp op) {
 Value tryFoldCastChain(CastOp op) {
   CastOp head = op, tail = op;
 
-  while(op) {
+  while (op) {
     if (!isIntOrBoolCast(op))
       break;
     head = op;
@@ -2259,6 +2259,7 @@ ParseResult cir::FuncOp::parse(OpAsmParser &parser, OperationState &state) {
   auto noProtoNameAttr = getNoProtoAttrName(state.name);
   auto visibilityNameAttr = getGlobalVisibilityAttrName(state.name);
   auto dsolocalNameAttr = getDsolocalAttrName(state.name);
+  auto annotationsNameAttr = getAnnotationsAttrName(state.name);
   if (::mlir::succeeded(parser.parseOptionalKeyword(builtinNameAttr.strref())))
     state.addAttribute(builtinNameAttr, parser.getBuilder().getUnitAttr());
   if (::mlir::succeeded(
@@ -2290,6 +2291,9 @@ ParseResult cir::FuncOp::parse(OpAsmParser &parser, OperationState &state) {
   if (parser.parseOptionalKeyword(dsolocalNameAttr).succeeded())
     state.addAttribute(dsolocalNameAttr, parser.getBuilder().getUnitAttr());
 
+  if (parser.parseOptionalKeyword(annotationsNameAttr).succeeded())
+    state.addAttribute(annotationsNameAttr, parser.getBuilder().getUnitAttr());
+
   StringAttr nameAttr;
   SmallVector<OpAsmParser::Argument, 8> arguments;
   SmallVector<DictionaryAttr, 1> resultAttrs;
@@ -2508,6 +2512,12 @@ void cir::FuncOp::print(OpAsmPrinter &p) {
   else
     function_interface_impl::printFunctionSignature(
         p, *this, fnType.getInputs(), fnType.isVarArg(), {});
+
+  if (mlir::ArrayAttr annotations = getAnnotationsAttr()) {
+    p << " ";
+    p.printAttribute(annotations);
+  }
+
   function_interface_impl::printFunctionAttributes(
       p, *this,
       // These are all omitted since they are custom printed already.
@@ -2517,7 +2527,8 @@ void cir::FuncOp::print(OpAsmPrinter &p) {
        getGlobalDtorAttrName(), getLambdaAttrName(), getLinkageAttrName(),
        getCallingConvAttrName(), getNoProtoAttrName(),
        getSymVisibilityAttrName(), getArgAttrsAttrName(), getResAttrsAttrName(),
-       getComdatAttrName(), getGlobalVisibilityAttrName()});
+       getComdatAttrName(), getGlobalVisibilityAttrName(),
+       getAnnotationsAttrName()});
 
   if (auto aliaseeName = getAliasee()) {
     p << " alias(";