Frequently Asked Questions on TAPENADELast modified: Tue Apr 10 10:36:09 CEST 2012

by the developers of Tapenade

We encourage you to subscribe to the "tapenade­users" mailing list to share with other users, to get the latest news, and tosubmit your questions and remarks !

Problems with the web interface : Problems with web browsers (Macintosh) Multiple sessions in parallel Approximative source­code correspondence

Problems related to Install : Download Tapenade Old versions of Tapenade System errors of the "not found" sort JAVA Version required Tapenade and Windows Tapenade, Windows and cygwin problem Tapenade with C programs on Windows, problem with cpp

Differentiation crashes : Directory separator linux/windows Languages accepted by Tapenade parsers, Syntax errors Syntax of directives that modify differentiation ($AD ...) JAVA errors: NullPointerException, ArrayOutOfBoundsException, etc System errors of the "out of memory" sort Errors on differentiated declarations Errors on complicated statements

Differentiated program doesn't compile : Missing differentiated routines for intrinsic or Black­Box routines

Differentiated program crashes : Divisions by Zero and NaN's PUSH/POP mechanism Wrong external's Some cases of "segmentation fault"

Returned Derivatives are wrong (or just need to be validated): Basic and extremely useful code preparation before differentiation Validation of the differentiated programs

Functionnalities : Building the context for calling the differentiated code Copied procedures in the differentiated code Too many Warning and Error messages Includes and Comments Type mismatches Aliasing Language accepted by the A.D. engine ; Limitations Black­box procedures, unknown external procedures Domain of validity of derivatives Original statements and results disappearing from the Reverse code Differentiating multiple top/head procedures, Differentiating Libraries Independent and Dependent sets. Initialization and usage of the differentiated parameters Default sizes of basic data types Parameterized array sizes Multi­directional modes (i.e. option ­multi) Getting the Jacobian using the Multi­directional mode Second Derivatives and Hessians

Performance Improvements: Controlling checkpointing in the reverse mode Independent Iterations Loops ($AD II­LOOP) Binomial Checkpointing ($AD BINOMIAL­CKP) User­defined additional checkpoints ($AD CHECKPOINT­START and $AD CHECKPOINT­END)

Linear Solvers

Documentation : Acronym All Warning or Error messages Other Differentiation Modes Other AD tools Differences with Odyssée Documentation, Archives of the Mailing list Sending remarks, questions, and bug reports

Acronym : What does TAPENADE mean? Why did you choose this name?

Well, we are all searching for the ultimate Automatic Differentiation tool, but we would be silly to claim that we found it! Therefore,TAPENADE could mean "Tangent and Adjoint PENultimate Automatic Differentiation Engine".

Honestly, we arranged this acronym after we chose the name! We simply liked the name TAPENADE because it is a typicalingredient of our local cuisine. It is made with capers, olives, and anchovies, and it makes a perfect apéritif in summer, on smalltoasts with a glass of cold rosé wine. Just in case you couldn't find some in your local grocery, here is a recipe of Tapenade from afamous book of provençal cuisine.

Languages accepted by Tapenade parsers, Syntax Errors : After calling for differentiation of my files, I get the"Something went wrong during differentiation" web page, which tells me that there is a syntax error in my files.What can I do ?

Yes, this happens in many cases. First notice that there are many sorts of so­called "syntax errors", which result in slightly differenterror messages. You may get one or many of the following:

Syst: ff (No such file or directory) Syst: Fortran Parser: ff No such file or directory: Interrupted analysis :Some required file was not found. Typically an include file was not given.Tool: Unknown suffix file ff Tool: No parser for ff. File skipped :TAPENADE tries to guess the language of the file from its suffix. Then it looks for a parser for this language. Either step mayfail.File: Fortran Parser: Lexical error at line nn in file ff, ... : There is some character or word atthis line nn which is not understood in the current file's programming language (e.g. Fortran).File: Fortran Parser: Syntax error at line nn in file ff :At this line is some syntax construction which is not permitted by the current file's programming language (e.g. Fortran).Syst: Unexpected source operator: op (xx): Interrupted analysisSyst: Unexpected source list: ops (xx): Interrupted analysis : This internal error says that one operator of the current file's programming language was forgotten in a preliminary stage ofthe tool. Please send us this error message so that we can fix it quickly.Syst: Not a tree operator: text : The parser has failed, and sent an incorrect tree to the differentiator. This is probably an internal error, because the normalmessage should be the well known "Syntax error" above.File: The file does not contain a top procedure : strangely enough, this message may appear as a resultfrom a syntax error. Syntactically correct files must declare subroutines or functions. If, for any reason, no routine can befound, differentiation cannot start.File: Fortran Parser: Lexical error at line 1 in program.f, Unknown character 'm' incolumn 1 to 5 : If your program uses the free format style and the file's suffix is .f or .F, you must select fortran95 asinput language with the ­inputlanguage fortran95 option in order to parse it without syntax error.

First of all, make sure that there are no missing or empty files. Then check that each file's suffix corresponds to its actual language:

.f or .F for Fortran,

.f90, .F90, .f95, or .F95 for Fortran90 or Fortran95,

.c or .C for COtherwise this is a "real" syntax error.

Our Fortran parser accepts all of the Fortran77 standard, plus most of the usual constructor's extensions, present or past (vax,sun, gould, connexion­machine, cray). Beware of possible conflicts between the standard and the extensions! For example in thestandard, white­spaces are not significant: if the text functional is found in the code, it is seen as the keyword functionfollowed by the identifier al, probably leading to a syntax error ! Therefore you mustn't call one of your code identifiersfunctional or subroutines. We couldn't change this rule because some old programs did strip away white spaces! Checkthat your file really complies with the standard or the constructor's extension. If this is the case, Tapenade should parse it correctly.If it does not, there is probably a bug in our parser. Please send us a message so that we can fix it quickly.

Our Fortran95 parser accepts of course Fortran90, and also the HPF extensions. Be careful with comments delimitor in thefree format style: comments must start with a "!". Our C parser accepts ANSI C plus a few common extensions, BUT it doesn'taccept C++ input.

If you really think your program complies with these standards, and our parser rejects it, please send us a message so that we canfix our parser. Notice furthermore that our parser does not support the cpp directives. For instance, we cannot parse filescontaining #define or #ifdef directives. Please run your files through cpp before sending them to Tapenade.

If the error you get is an internal error, please send us a message so that we can fix it !

Syntax of directives that modify differentiation ($AD ...): I get the following error when Tapenade parses my source:"File: Fortran Parser: Lexical error at line nn in foo.f95, Unknown keyword : c$adnocheckpoint"More generally, what is the exact syntax of the directives that I can use in my source to alter its differentiation?

Tapenade understands a small number of directives that can be used to alter the differentiated result. These directives must beprovided in the source code as single­line comments that begin with $AD. Since the syntax of comments changes with thelanguage, the syntax of tapenade directives changes accordingly, e.g. for directive "nocheckpoint":

In Fortran 77 :

C $AD NOCHECKPOINT

The C must be in column zero. There can be any number of white space (or none) before the $. Case is insignificant.

In Fortran 90­95 :

! $AD NOCHECKPOINT

The ! can be in any column. There can be any number of white space (or none) before the $. Case is insignificant.

In "not really ANSI­" C :

// $AD NOCHECKPOINT

The first / can be in any column. There can be any number of white space (or none) before the $. Case is insignificant but wesuggest you use upper case for a better visibility.

In C :

/* $AD NOCHECKPOINT */

The initial / can be in any column. There can be any number of white space (or none) before the $. Case is insignificant butwe suggest you use upper case for a better visibility.

The correct location of a directive in the source depends on the actual directive. At the moment, the available Tapenade directivesare:

$AD NOCHECKPOINT, described here$AD II­LOOP, described here$AD BINOMIAL­CKP, described here$AD CHECKPOINT­START, described here$AD CHECKPOINT­END, described here

Language accepted by the A.D. engine ; Limitations: I can see that Tapenade differentiates Fortran77 programs. What about Fortran95 or C ? What about their various constructors'dialects ? Do you have plans for object­oriented languages ? Do you differentiate programs with pointers, with records, withmodules?

We are talking here of actual differentiation, not parsing. About parsing, see the "Syntax" question. We tried to differentiate correctlythe standard features of Fortran (77 and 95) and of C, plus most of the usual constructor's extensions, present or past (vax, sun,gould, connexion­machine, cray). As far as differentiation is concerned, most of these extensions have little effect anyway. Ofcourse there are possible bugs and a few restrictions that will be lifted progressively. We do not differentiate pure C++ constructs(Tapenade doesn't even parse them!). In the following list, parts in black are what Tapenade should differentiate correctly, parts inred are what is not differentiated:

Fortran 95 free source form : our Fortran95 handles it. Be careful with comments delimitor in the free format style:comments must start with a "!"modules, BLOCKDATA : accepted and differentiated.overloading : accepted and differentiated.keyword arguments and optional arguments : accepted and differentiated.derived types, structured types : accepted and differentiatedSWITCH/CASE statements : accepted and differentiatedpointers : accepted, analyzed and differentiated except in some cases in reverse modedynamic allocation, malloc's, free's : accepted and differentiated in tangent mode. In reverse mode, there are somelimitations (that we must document some day) e.g. corresponding allocations and frees must live in the same procedure. Inany case, we believe these constructs should preferably be kept out of the computational kernel, which should contain thefragment to differentiate.

namelist : accepted and differentiated.vector notation, WHERE and FORALL constructs, mask's, SUM's and FORALL's : accepted and differentiated.argument passing call­by­value, call­by­value­result, call­by­reference : accepted and differentiated.interleaving declarations and instructions, declarations with initializations : accepted and differentiated. The attachedcomments are approximatively re­placed at their correct positions.C if­expressions, e.g. x = 1 + (y<0?­y:y); : parsed but incorrectly differentiated, especially in reverse mode.array constructors e.g. A(:) = (/X(2)­X(1),Y(2)­Y(1)/) : arrayConstructor are parsed but incorrectly differentiatedin reverse mode.

Specifically for the reverse mode: a program piece that will be checkpointed (e.g. a procedure call) must be reentrant. This meansthat there must be a way to call it twice, at the cost of some manipulations to restore the initial state. For instance a procedure thatmodifies an internal hidden variable or state (think of IO pointers in files, or private counter variables...) is not reentrant. In this case,you must either transform the program piece to make it reentrant, or tell Tapenade not to checkpoint it.

Differentiation of object­oriented languages is still not in our immediate goals, and neither are the templates of C++.

Black­box procedures, unknown external procedures: Tapenade complains about unknown external or intrinsic procedures, how can I teach it about them ?I need to define a new set of library files or modify the existing *Lib files.What is the syntax in there and what does it mean?

If the program you want to differentiate contains calls to EXTERNALs, INTRINSICs, library, or other "Black­Box" routines, then you mayneed to provide TAPENADE with some information about these Black­Box routines. You may also have to do this if we forgot to add intothe standard library files the information on some intrinsic subroutines that you actually use.

It can also happen that you have the source of a procedure, but you don't want to give it to TAPENADE, because TAPENADE would thendifferentiate it in its own standard manner, and you know how to differentiate it better by hand! So make it a Black­Box by just forgetting toprovide its source.

Some intrinsic functions, such as MAX, are not well treated when they have an unusual number of argument. For instance, MAX(x, y+z,0.0) will not be analyzed, nor differentiated corectly. The workaroud is either to replace this by two nested calls to MAX, or to treat the 3­arguments MAX as another Black­Box function.

So, for whatever reason, TAPENADE has no source for a subroutine or function (suppose it is called BBOX). This has some unfortunateconsequences. First, during type­checking, TAPENADE cannot compare the number and types of the actual arguments to the numberand types of the formal arguments of BBOX. Furthermore, it doesn't know which arguments will be read and/or overwritten. Morespecifically for AD, TAPENADE doesn't know how to propagate activity through BBOX, i.e. which outputs of BBOX depend in adifferentiable manner on which inputs of BBOX. Last but not least, TAPENADE doesn't know the name of the differentiated version ofBBOX, in the case where you have programmed it already.

Notice that, although all this information is missing, TAPENADE will often generate good enough code, thanks to a number ofconservative assumptions:

The types and the number of arguments are deduced from the actual usages. If these usages don't match, there will be an errormessage.By default, a black­box routine is supposed to communicate with the calling code only through the calling arguments: TAPENADEsupposes that the black­box routine doesn't use arguments through COMMON's and other globals. Warning: this assumption isnot conservative: if the black­box routine actually refers to a global and nothing is done to tell TAPENADE about that,TAPENADE may produce wrong code!All actual arguments are supposed to be possibly read inside the black­box routine, except of course the returned result of afunction.All actual variable­reference arguments are supposed to be possibly overwritten inside the black­box routine if it is an external or anintrinsic procedure. If it is an intrinsic function, we suppose no argument is overwritten.Any variable argument of a differentiable type is assumed to return an output value that depends (in a differentiable way) on theinput value of every argument of a differentiable type.The differentiated routine is supposed to have the same name and shape as what would have been created by TAPENADE, i.e.BBOX_D or BBOX_B.

If you want to override the assumptions above, allowing Tapenade to produce better code, you may provide some knowledge of theBlack­Box routines that are used in the program. The *Lib (or the obsolescent *ADLib files) are one way to give it this knowledge. Anotherway is to write dummy routines.

SPECIFYING BLACK­BOX ROUTINES IN *Lib FILES:

The "*Lib" files contain information about library, intrinsic, external routines or any other Black­Box routine. This will help Tapenadeproduce more efficient code. For instance, there is one such file "F77GeneralLib" already in you installation "lib/" directory. You maymodify it. You may also create your own files such as "MyAppliGeneralLib", in any "MyDirectory", and then you will pass these filenames to your tapenade command using the ­ext option.

tapenade ­ext MyDirectory/MyAppliGeneralLib ...

Each entry is about one subroutine or function, and gives info about number, type, intent of arguments, and possibly info related to AD.For example you might type, for a subroutine named BBOX and for a function called FBBOX:

subroutine bbox:

external: shape:(param 1, param 2, common /c1/[0,*[, common /c2/[0,16[, common /c2/[16,176[, param 3, param 4, param 5) type :(ident real, ident real, arrayType(ident real,dimColons(dimColon(none(),none()))), modifiedTypeName(modifiers(ident double), ident real), arrayType(modifiedTypeName(modifiers(ident double), ident real), dimColons(dimColon(none(),none()))), ident boolean, arrayType(ident character,dimColons(dimColon(none(),none()))), ident character) ReadNotWritten: (1, 0, 0, 0, 0, 0, 1, 1) NotReadThenWritten: (0, 0, 1, 0, 0, 0, 0, 0) ReadThenWritten : (0, 1, 0, 1, 1, 1, 0, 0) deps: (id, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, id, id)

function fbbox: intrinsic: shape: (param 1, param 2, result) type: (ident real, ident real, modifiedTypeName(modifiers(ident double), ident real)) ReadNotWritten: (1,1,0) NotReadThenWritten:(0,0,1) NotReadNotWritten: (0,1,0) derivative: binary(call(ident fbbox, expressions(metavar X, metavar Y)), mul(realCst 2.0, metavar X), expressions(binary(metavar X, none(), add(intCst 5, metavar Y)), binary(metavar Y, none(), div(realCst 1.0, metavar Y))))

The "external:" line states that the procedure is external. For an intrinsic, just replace this line by "intrinsic:".The "shape:" part is necessary, because it defines the (arbitrary) order in which arguments will be referred to in the sequel. Its syntax iseither param N that designates the N­th formal parameter, or result for the returned value if BBOX is a function, or common/xx/[0,8[ for an argument which is found from offset 0 to offset 8 in common /xx/.All next lines are optional. If they are missing, the corresponding conservative assumptions are made.The "type:" part gives the types of each argument, in the same order. The syntax of a type is rather heavy: we didn't find the time tomake it cleaner yet, sorry! Actually it's the internal tree representation of the type inside Tapenade memory. In BNF­like, a TYPE can be:

TYPE ::= ident NAME(NAME belonging to real, complex, integer, character, boolean, for primitive types)

TYPE ::= modifiedTypeName(modifiers(MODIFIERS), TYPE)(MODIFIERS being a list of mostly ident and intCst such as double or 16 for primitive type size modifiers)

TYPE ::= arrayType(TYPE, dimColons(DIMS))(for arrays with a fixed number of dimensions)

DIMS ::= DIMDIMS ::= DIM , DIMS

DIM ::= dimColon(BOUND, BOUND)BOUND ::= none()(for an unspecified dimension lower or upper bound)BOUND ::= intCst N

TYPE ::= arrayType(TYPE, dimColons())(for arrays with any number of dimensions)

TYPE ::= pointerType(TYPE)(for a pointer to another type (available in near future))

TYPE ::= void()(void type, mostly means you don't want to care about it)

For example, real*8 is written modifiedTypeName(modifiers(intCst 8), ident real)Arrays are built with the combinator "arrayType", that binds the array elements' type with a list of dimensions ("dimColon") which arepairs of the lower and upper bound. So far, these bounds are not very useful, so you might just as well leave them as "none()" or even

use the arrayType(TYPE, dimColons()) form. Take a look at F77GeneralLib to find more examples.This mechanism is far from being complete. For example, there is still no syntax for structured argument types ("derived types"), althoughthey are handled well inside Tapenade. Nothing exists for named arguments, nor for a variable number of arguments. For a variablenumber of argument, you may have to create a *Lib entry for each number of arguments that actually occurs in your code.

In addition to this, there is our "metavar" type:

TYPE ::= metavar NAME

This metavar type represents any type. If it appears with the same NAME several times in the list of types of the present procedure, eachoccurrence must match the same type. The value of NAME just indicates which "sort" of type should be there, i.e. if actual pattern­matching fails to find a solution type for the metavar, then TAPENADE will choose ident NAME. Otherwise you may name your metavar"foo" instead of "real", and it should work just the same. This mechanism is by no means complete, but it helps.Look at function "SUM" in F77GeneralLib for a less trivial use of metavars: in SUM, the metavar lets you specify that it takes an array ofREAL and returns a REAL, or takes an array of REAL*8 and returns a REAL*8, or even an array of INTEGER and returns an INTEGER.

The "ReadNotWritten, NotReadThenWritten, ReadThenWritten, NotReadNotWritten" info gives the read­writesignature of each argument. There is 1 for one argument when there exists at least one execution case of the procedure for which at leasta part of this argument is "affectedInThisWay", where "affectedInThisWay" is the name of the info (i.e. one of ReadNotWritten,NotReadThenWritten, ReadThenWritten, NotReadNotWritten). This is a new notation, valid from version 3.6 on. This newnotation replaces the previous notation (R, NR, W, NW) which had a different meaning, less accurate. Although the previous notation isstill accepted, we advise you to switch rapidly to the new notation in your home­made libraries.

The "deps:" entry is related to AD. It is the differentiable dependency matrix, of each output value with respect to input values, for eachargument. The matrix is square (id elements count for one row), and the order of the rows is the order of the parameters as listed in the"shape:" entry for this procedure. The same holds for the order of the columns. For example, the element in row 2, column 5 specifieswhether the output value of param 2 depends on (i.e. "has a non trivial­zero derivative with respect to") the input value of the variablefrom offset 16 to 176 in common /c2/. Here there is a zero, meaning a trivial zero derivative, i.e. no dependency. For example here, theresult in parameter 2 depends on the input parameters 1 and 2, the result in the first variable in common /c2/ doesn't depend on anyinput, and the result in the tail array of /c2/ depends on the inputs in parameter 2 and in the whole of /c2/. An "id" entry instead of acomplete line states that the argument is left unmodified, which is a stronger information than a line with just one "1", because it impliesthat the partial derivative is certainly "1.0". This is the case here for parameters 1, 4, and 5. On the other hand, parameter 3 is overwritten,but of course depends on nobody, since it has a non­differentiable type. For the same reason, nobody depends on parameters 3, 4, and 5.Notice also that nobody depends on the input in /c1/: it is an output only.

The other thing you may want to specify is the actual derivative of the black­box procedure. Three main possibilities:

It may happen that after activity analysis, TAPENADE finds out that the differentiated program does not actually need to differentiatethe black­box. In other words, no argument of the black­box is active. Then you're very lucky, there is nothing you need to do!If the routine really has active arguments, then it may happen that it is actually a function, and a simple one so that you can definethe partial derivatives of its output with respect to each input, as a simple cheap expression. This can include calls to anotherintrinsic. TAPENADE can use those partial derivatives both in tangent and reverse mode. Think for example of COS(X). There'sonly one partial derivative you need to specify which is the partial derivative of the result wrt the first (and only) argument, and thisderivative is ­SIN(X). To specify this, the "derivative:" entry in the "*GeneralLib" file gives a pattern that must match theoriginal call, and patterns that defines each partial derivative. Here again, the syntax is ugly because we didn't find the time to cleanit yet. Sorry again! Let's recall what we wrote for the function fbbox above:

function fbbox derivative: binary(call(ident fbbox, expressions(metavar X, metavar Y)), mul(realCst 2.0, metavar X), expressions(binary(metavar X, none(), add(intCst 5, metavar Y)), binary(metavar Y, none(), div(realCst 1.0, metavar Y))))

The parts in green are completely meaningless, but must be respected because TAPENADE expects to find them there.In this example, one can find, in order, the pattern that must match the actual call, therefore instantiating the metavariables X and Y,then an expression which is a common factor to all partial derivatives (which can be "none()" if there is no common factor), thenthe list of each partial derivative of the function's result with respect to each metavariable: for each metavariable key, there is anexpression that defines the corresponding partial derivative expression, which has to be multiplied by the common factor if present.For instance in the above example, we have expressed that the partial derivative of FBBOX(X,Y) with respect to X is theexpression 2.0*X(Y+5) and its partial derivative with respect to Y is the expression 2.0*X/Y. Notice that we didn't give a"deps:" entry, because the default for an intrinsic function is just what we want! More examples can be found in theF77GeneralLib file, for instance the specification of the partial derivatives of COS mentioned above.If there is no way to define the partial derivatives explicitly, then you must write and provide the derivative code of the black­boxroutine for the current differentiation mode (e.g. tangent, adjoint,...). Follow the message issued by TAPENADE to know whichinputs (resp. outputs) of the black­box must be considered active. Follow the syntax that TAPENADE has chosen when generatingthe calls to this differentiated black­box in the differentiated code. Then simply write these differentiated external routines by hand ina separate file, using your preferred method/algorithm. When everything else fails, you may even use divided differences, always atthe cost of some accuracy. Finally link the resulting code with the rest at the end of compilation time.

Please notice that these *Lib files are not able to represent all you would like to. For example, there is no support for procedures withvariable number of arguments or named arguments. We hope it can be useful in most situations, though, until we devise a better and morecomplete syntax for these *Lib files.

SPECIFYING BLACK­BOX ROUTINES WITH DUMMY ROUTINES:

Sometimes the *Lib files cannot express some behavior of the black­box procedure, Tapenade does not understand this information well

and doesn't differentiate the black­box procedure correctly. Sometimes also, the syntax of the *Lib files is really painful and one looks for asimpler method. Here is a possible simpler method that you can use. Write a dummy definition of the problematic black­box procedure,just like a standard procedure, with statements that just "represent" the data dependences of the actual procedure. We mean that thecomputations inside the dummy procedure may be totally wrong and meaningless, provided they implement the same differentiable data­dependencies between the inputs and the outputs. In other words again, for each output z that depends in a differentiable way on inputs,say, x and y, just write a statement like

z = x*y

in the dummy procedure. Give this dummy definition only to the differentiation process, and remove the differentiated dummy procedurefrom the differentiated code returned by Tapenade. Finally, write by hand the correct differentiated black­box procedure and link it with thedifferentiated code returned by Tapenade.

Other Differentiation Modes: I need higher­order derivatives, Hessians, Taylor series.I need the full Jacobian Matrix.

In its present form, Tapenade builds programs that compute directional derivatives and gradients, i.e. in our jargon the forward andthe reverse mode of A.D. Tapenade does not provide Jacobians, nor higher­order derivatives, nor Hessians, nor Taylor series.

However, there are ways to cope with that. For example, one can use the forward mode twice, to get directional second derivatives.We know of some people who have tried that with Tapenade, and apparently it worked. Notice however that this is certainly lessefficient than a specific mode for second derivatives. One reason for that is that if you use the forward mode twice, you get tworedundant copies of each computation of the first derivatives!

Then, from the higher­order directional derivatives, you can derive the higher­order Hessian tensors. This is sometimes more cleverthan computing these tensors from scratch, because one may take advantage of the tensor's symmetries. This question isdiscussed for example in GriewankUtkeWalther97

Tapenade does nothing about Taylor series nor intervals. These two could be interesting new modes to include into Tapenade, butthis is not in our current plans.

Many applications need the Jacobian matrix. Here are some ways to get it:

There could be a special purpose differentiation mode, propagating a partial Jacobian throughout the execution of the originalprogram. This was implemented once into Odyssee, and the results were interesting, though slightly disappointing. Thismode was not maintained in the next version of Odyssee (1.7). However there is a potential source of optimization comingfrom array analysis in regular loops, as explored in TadjoudineEyssetteFaure98One can compute it column by column, by repeated executions of the forward mode, one execution for each direction in theorigin space. i.e. for each element of the Cartesian basis of the input space. Actually fewer directions are required when theJacobian matrix can be shown to be sparse enough (see the litterature, e.g. the Griewank­Walther 2008 book [GrieWalt08]).This can be improved by computing the derivatives for many directions in the same run. This is the so­called vector mode ormulti­directional mode, first found in ADIFOR and available in Tapenade. Multi­directional mode can exhibit someparallelism, as shown in BuckerLangMeyBischof01. As compared to ADIFOR, Tapenade brings the novelty that differentiatedinstructions, which are now loops, are gathered using a special data­dependency analysis. This reduces the overhead due tothe numerous one­instruction loops.One can compute it line by line, by repeated executions of the reverse mode. Due to the intrinsic difficulties of the reversemode, this is recommended only when the number of inputs is larger than the number of outputs, but on the other hand thiscan benefit from specific improvements of the reverse mode such as a bettter detection of common sub=expressions inderivatives. At the present time, Tapenade doesn't provide a multi­directional mode for the adjoint mode, but this couldchange if more applications would use it.

Type mismatches: Tapenade Type­Checker sends me a lot of messages of the form:(TC16) Type mismatch in assignment: REAL*8 receives DOUBLE PRECISIONWhat do they mean? Are they important? What must I do?

These messages may be important, but in many cases they are only "paranoiac" warnings. Your compiler may perfectly well acceptthat an argument of kind DOUBLE PRECISION is passed to a procedure that expects a REAL*8 or vice­versa. However this isagainst the Fortran standard, and it may be the cause of portability problems in the future. This is why Tapenade complains whileyour compiler doesn't.

For instance g77 does not complain on:

real*8 varvar = 1.0d0

but Tapenade does because the standard says that 1.0d0 is DOUBLE PRECISION and not REAL*8.

More importantly concerning AD in the reverse mode, the PUSH and POP procedures ­­ that are used intensively ­­ rely on an exactknowledge of the size in bytes of their argument. So at the minimum you should check that the sizes of these types match on themachine/compiler you will use, and that these are the sizes that Tapenade will use. To check the sizes on your machine/compiler,refer to the beginning of the FAQ section on validation of the differentiated programs . To modify the sizes used by Tapenade toadapt them to the results of the above check, use the command­line options ­r8 etc.

Anyway, all these precision problems are a nigntmare. We agree that most of the messages that Tapenade sends about this are a

nuisance. Most probably you can neglect them. Maybe some day we shall offer an option to hide them away.

Nevertheless it still occurs sometimes, especially with Fortran95, that these messages indicate bugs in Tapenade. For instanceTapenade may have lost a dimension of an array. In this case, it is unlikely that differentiation will work. In this situation, when yousee that your program is obviously correct and Tapenade complains, please send us a message so that we can improveTapenade.

Aliasing problems: I get the following message from Tapenade:(DF02) Potential aliasing in calling function Func, between arguments argumentsWhat does it mean and is it important?

This means that Tapenade detected a memory overlap between arguments of ranks n1 and n2 of a call to a subroutine or function.

What is aliasing? Suppose a subroutine has many arguments (i.e. formal parameters or commons or globals...) At each call site,each of these formal parameters is given an "actual parameter", i.e. a reference to an actual variable, scalar or array, or anexpression, or a constant... Aliasing happens when the same actual parameter is given to two different formal parameters. and oneof these two formal parameters is overwritten by the subroutine. More generally, it happens when two actual parameters overlap inmemory, and at least one may be overwritten. Please notice that such aliasing results in code that does not conform to the Fortranstandard. e.g. see MetcalfReid96 section 5.7.2 page 91. This is an important warning, because Tapenade, like most static analysistools, analyses each subroutine in its own local context, i.e. assumes that each formal parameter is a different memory location.And this assumption leads to a generated program (differentiated) that will fail if there is aliasing. For example:

subroutine F(real a, real b) a = 3*b

would be differentiated in the reverse mode as (grossly) :

subroutine F_B(real a, real ab, real b, real bb) bb = bb + 3*ab ab = 0.0

and this works fine as long as the actual a and b are different variables. But if a=b=x, F does x = 3*x, and what you want for F_Bis xb = 3*xb. If you check, this not what you get by calling the above F_B. Similar problems arise in most programtransformations, such as parallelization or partial evaluation, or others...

Notice however that aliasing, here detected by a static analysis, is by essence undecidable. Therefore Tapenade may detectaliasing when there is none! Also, it is frequent that, although there is aliasing, the differentiated program will run fine. It is theresponsibility of the end­user, who knows the program, to check that his aliasing is harmless. Usually, when aliasing is really aproblem, one can easily fix that by introducing temporary variables, so that memory locations of the parameters do not overlap.

Missing differentiated routines for intrinsic or Black­Box routines: I can't compile my code because it misses the differentiated routines for some intrinsics, externals, or Black­Box routines.I get a compiler message saying it needs to link with DCMPLX_BHow do I define the adjoint of an intrinsic routine?

This is perfectly natural.

In your original code were Black­Box routines, such a DCMPLX, i.e. either INTRINSICs or EXTERNALs or other routines for whichyou didn't provide the source. As a consequence, Tapenade has placed in the differentiated code a call to the differentiated routineDCMPLX_D in tangent mode or DCMPLX_B in adjoint mode, and sent you a message requesting you to define these new routines.Now is the time to do it.

Take the example of DCMPLX with 2 arguments:

cc = DCMPLX(rr,ii)

This call builds a DOUBLE COMPLEX number with real part rr and imaginary part ii, and puts the result into variable cc.

The variables on which DCMPLX operates are rr, ii and cc. To consider only real numbers, let's split cc into its real andimaginary parts cc.r and cc.i. Since rr and ii may be used later, we consider them also as (unchanged) outputs of DCMPLX.Similarly cc can be seen as an (unused) input. All in all, DCMPLX operates from R4 to R4, with some conventional ordering e.g.:

(rr ii cc.r cc.i)

We can thus write the Jacobian matrix of DCMPLX, which is in this case simply:

(1 0 0 0) (0 1 0 0) (1 0 0 0) (0 1 0 0)

as the output cc.r receives exacty rr, the output cc.i receives exacty ii, and neither rr nor ii are modified.

The tangent differentiation of DCMPLX just implements the matrix­times­vector product with this Jacobian, i.e.

(rrd ) (1 0 0 0) (rrd ) (iid ) (0 1 0 0) (iid ) (ccd.r ) = (1 0 0 0) * (ccd.r ) (ccd.i ) (0 1 0 0) (ccd.i )

hence the definition of DCMPLX_D you must provide:

function DCMPLX_D(rr, rrd, ii, iid, cc) DOUBLE PRECISION rr,rrd,ii,iid DOUBLE COMPLEX cc, DCMPLX_D DCMPLX_D = DCMPLX(rrd,iid) cc = DCMPLX(rr,ii) end

The adjoint differentiation of DCMPLX just implements the matrix­times­vector product with the transposed Jacobian, i.e.

(rrb ) (1 0 1 0) (rrb ) (iib ) (0 1 0 1) (iib ) (ccb.r ) = (0 0 0 0) * (ccb.r ) (ccb.i ) (0 0 0 0) (ccb.i )

hence the definition of DCMPLX_B you must provide:

subroutine DCMPLX_B(rr, rrb, ii, iib, ccb) IMPLICIT NONE DOUBLE PRECISION rr,rrb,ii,iib DOUBLE COMPLEX ccb rrb = rrb + DBLE(ccb) iib = iib + DIMAG(ccb) ccb = DCMPLX(0.d0) end

The above equation is a "vectorial" assignment in R4, which is done "in parallel". Therefore we were careful not to reinitialize ccbbefore it is used to update rrb and iib. Also, since this is adjoint mode, it is no use to return the value of cc

Divisions by Zero and NaN's: The differentiated program returns NaN's in the derivative variables. However my original program is clean and doesn't returnNaN's !

Differentiation can introduce NaN's in the resulting program, because programs are sometimes non differentiable. (You may takea look at the FAQ item about the domain of validity of derivatives) In general, AD is in trouble when it comes to differentiating afunction at places where it is not differentiable.

In this situation, we prefer that the differentiated program return NaN, which at least shows that something went wrong. This isprobably better than silently returning a wrong derivative that may lead to other wrong results later. The end­used can then analysethe original program, and maybe transform it to make it differentiable.

However there are situations where Tapenade can do a slightly better job, automatically. Non­differentiablility often occurs forSQRT, LOG, and exponentiations. For SQRT, we can do something a little more clever than returning NaN, but only in one particularcase:Suppose instruction is

a = SQRT(b)

the tangent derivative instruction is

ad = bd/(2.0*SQRT(b))

that returns NaN if b is 0.0. But if bd is also 0.0 at this time, then we feel it is safe to return

ad = 0.0

On the other hand, we think it is not safe to test on b or a being 0.0 or close to 0.0. The reason is that in this case the derivative isreally undefined, so it's up to the user to do something about it. It would be dangerous to return ad = 0.0 automatically. Only theuser can do it by hand, and check that it "works" for the current application.

Something similar is done for exponentiations. Nothing is done for LOG. We should probably do the same as for SQRT!

A similar reasoning applies to the reverse mode: Tapenade generates statements that detect special cases and build 0.0derivatives.

However there is a weakness in Tapenade when it comes to returning Nan's: unlike what happens in Adifor, there is no warningmessage printed at the moment when the Nan appears. Therefore, in the very frequent case where the compiled code does notstop when the Nan appears, computation goes on as usual and it is very difficult to find the place where the problem originated. Weare aware of this problem and this should be improved.

NaN's can also appear for a slightly different reason: for some given inputs, some functions have an output which is large, but notyet in overflow, whereas the derivative is definitely in overflow. Think of 1/x, whose derivative is ­1/x**2. Again this requires aspecial treatment. One way could be to switch to double precision.

We recently ran into a very annoying NaN problem related to uninitialized variables. Please avoid using uninitialized variables inyour code! Tapenade tries to send messages about uninitialized variables, but this is a typical undecidable question, so we cannotfind all of them. The following summarizes the problem that we found on a very large code...

subroutine TOP(x,y) real x,y real BAD(2) call SUB(BAD,x) y = BAD(1) end

subroutine SUB(BAD,x) real x, BAD(2) BAD(1) = x*x BAD(2) = BAD(2)*x end

You can see that array BAD is not initialized. BAD(2) may very well contain something bad like a NaN. However this has no visibleimpact on this original code because the resulting x and y are not polluted by the NaN. Still this code is going to silently computeon NaN's, and you don't want this. Now if we differentiate this code in reverse mode, the following statement appears in SUB_B:

xb = xb + bad(2)*badb(2)

and if the initial BAD(2) is a dirty NaN, then this computes NaN times 0.0, which on some compilers evaluates to NaN, and theresulting xb gets corrupted!

Domain of validity of derivatives: What does AD compute when the fonction is non­differentiable?Why is the Tapenade derivative always 0.0 for my dichotomy­based resolution program?To what extent can I trust the derivatives computed by Tapenade?Is there a discontinuity in my function close to the current input?How can I evaluate the differentiability domain around my current input?Is there any support in Tapenade for non­differentiable functions?

If the Tapenade­differentiated code is executed right on a non­differentiable input, it may happen that the differentiated programreturns NaN's as derivatives. See the FAQ item about divisions by Zero and NaN's.

Also, although it is hard to admit :­), there may be bugs in Tapenade :­( Therefore, you probably need to go through the validationprocess to make sure that the derivatives are correct, compared to Divided Differences. Although inaccurate and expensive,Divided Differences are robust and therefore return good enough derivatives for validation.

However, it may happen that the differentiated program just does not notice that it is on (or close to) a discontinuity. This is not abug of Tapenade; this comes from the nature of AD. Think of a dichotomy­based program that solves f(x,y)=0, returning y as afunction of x. The derivative of y with respect to x is in general nontrivial. Robust Divided Differences will actually return (anapproximation of) the correct derivative. But AD will return 0.0, and this can be explained:the dichotomy­based program actually sees f as a piecewise constant function, where each piece is of the size of machineprecision. In other words, for values of x that differ only very slightly, the value of y is exactly the same, and thus the derivative isrightly 0.0. Only this is not what you had in mind when you wrote the program!

Apart from the immediate rule of thumb which is "don't even think of differentiating dichotomy­based code, don't even write it!", theconclusion is that programming often introduces discontinuities into perfectly differentiable mathematical functions, and AD can't domuch about it (true, Divided Differences behave better on this question!). When this happens, the derivatives computed by AD arejust useless or plainly wrong. All we can do is warn the user when this happens, which is when there is a non­differentiability"close" to the current point, i.e. around the current input values.

In Tapenade, there is an experimental differentiation mode that evaluates the domain of differentiabillity around the current point,following a given direction in the input domain. The command­line argument ­directValid builds a special differentiatedprogram which, given a point in the input space and a direction in the input space (very much like in the tangent mode), evaluatesthe interval of differentiability along this direction. This uses subroutines called VALIDITY_DOMAIN_...(,), which are defined

in the file validityTest.f provided in the AD First­Aid kit. After execution of the differentiated program, the interval ofdifferentiability around the given input point and following the given direction is found in COMMON /validity_test_common/ gmin, gmax, infmin, infmaxwhere gmin (resp. gmax) is the lower (resp. upper) bound, and if there is no constraint on the lower (resp. upper) bound, theninfmin (resp. infmax) is .TRUE.

Caution: If, inside the same execution, you need to perform several independent calls to the "directional validity" differentiatedcode, for different inputs or for different directions, then don't forget to reset the validity interval to ]­infinity,+infinity[ between twosuccessive calls. Do this by resetting both infmin and infmax to .TRUE.

Using this mode in every direction of the Cartesian basis of the input space will give a box inside which derivatives and derivatives­based optimization are valid.

Differences with Odyssée : I have been using Odyssée in the past. What are the modifications from Odyssée to Tapenade ?

Details on this question in the history of successive versions.

TAPENADE is the successor of Odyssée. Therefore, TAPENADE is an incremental improvement to Odyssée. However, we had toradically modify the internal structures of Odyssée to take profit of compilation theory and technology. The internal representation ofa program is now a Call Graph, with one node per subroutine, and each subroutine is now a Flow Graph, whose nodes are BasicBlocks. Each Basic Block is equipped with a Symbol Table, and Symbol Tables are nested to capture scoping. Basic Blockscontain instructions, represented as Abstract Syntax Trees. Static analyses now run on this structure, and this yields an enormousspeedup in differentiation time, with respect to Odyssée. Right from the beginning, this internal representation of programs wasdesigned independently from a particular imperative language, like Fortran or C. For example data structures are already managed.In principle, adaption to C, say, just requires a C front­end and back­end, i.e. a C parser and a C pretty­printer.

About the differentiation model, TAPENADE is very similar to Odyssée. It still provides you with the forward and reverse mode ofA.D. There are some improvements though:

The activity analysis, a static analysis that finds the variables that need not be differentiated, is now bi­directional. It detectsthe variables that do not depend on the independents, and the variables that do not influence on the dependents. This leadsto fewer differentiated variables.We included the TBR analysis, a static analysis for the reverse mode. It finds variables which, although they are overwrittenduring the forward sweep, need not be saved because they will not be used during the reverse sweep.We implemented a new storage strategy for the reverse mode, named the global trajectory. This stores intermediate valueson an external stack rather than on static arrays that thad to be dimensioned by hand by the user of OdysséeSnapshots in the reverse mode are smaller, because they use In­Out analysis to find out variables that are really necessaryfor duplicate execution of the checkpointed piece of code.The inversion of the control, in the reverse mode, is done on the Flow Graph. This gives a better, more readable transformedsource, that the compiler can compile better.TAPENADE now provides a multi­directional forward mode, similar to the ADIFOR's vector mode. It computes the directionalderivatives in many directions simultaneously. Derivative loops are gathered using data­dependency analysis.Data­Flow analysis is used to optimize the transformed source, at the level of each Basic Block.

We provide two ways of using Tapenade: either by connecting to the web server, or by downloading it to install it locally. There is agraphical user interface based on XHTML, and a simple command­line interface.

Documentation, Archives of the Mailing list : What is the available documentation on Automatic Differentiation in general and specifically on Tapenade ? How can I take advantage of the experience from previous users of Tapenade ?

Here is a short introduction to "what is AD?"The web site of the AD community www.autodiff.org contains the most comprehensive documentation on AD.About Tapenade, apart from the tutorial here, you should start by looking at the Tapenade user's manual.Another good starting point is our Documents page.If you subscribe to the tapenade­users mailing list, you will have access to the archive of previous discussions at the address:https://lists­sop.inria.fr/wws/arc/tapenade­users.

PUSH/POP mechanism : Where are the PUSH and POP routines defined?

We provide you with the source of these routines, which comes as two files adBuffer.f and adStack.c. You may download itby clicking on the "Download PUSH/POP" button in the differentiation result page, or elseDownload First­Aid Kit

In some cases, the compiler may complain that some required PUSH/POP procedure for some unusual type is not inadBuffer.f, e.g. _pushinteger16. In this case edit adBuffer.f and either uncomment the required procedure it if it is there oror create it as indicated (see inside file adBuffer.f).

Original statements and results disappearing from the Reverse code : My original procedure computed an output Y whereas the reverse differentiated procedure does not return the correct value of Y.Some statements that are present in the original code have simply disappeared from the reverse code (this may even result in

empty loops or if branches).

By default, the reverse mode of Tapenade performs adjoint liveness analysis. This means that some statements from the originalprogram, which were "live" i.e. which computed something needed by the rest of the program, are not live anymore in the reversedifferentiated program. This often occurs around the end of a forward sweep, just before the backward sweep. The reason is: apriori the only result you expect from the reverse differentiated program is the derivatives. In particular the original result of theoriginal program is not considered as a necessary result of the reverse differentiated program. Therefore statements that areneeded only for this original result become dead code. More explanations and a small example are in the TAPENADE 2.1 user'sguide, specifically on page 33­34. A command­line option can toggle this mechanism off, see "­nooptim adjointliveness"on page 38.

For users who want the reverse differentiated program to preserve the original outputs of the original program, we advise first totoggle adjoint liveness analysis off. However this is not enough, because the PUSH/POP mechanism will naturally destroy severalof the program's original results. We advise that you modify the differentiated program by hand, saving the variables you want intotemporary variables at the end of the forward sweep, and restoring these variables at the end of the backward sweep.

Basic and extremely useful code preparation before differentiation I have a code I want to differentiate. How should I proceed?

Just "Differentiating a code" does not mean much, unless you know precisely which part of the code is the math function you wantto differentiate, which are its "independent" inputs and "dependent" outputs, and maybe which differentiation mode you want to use.We advise you to follow the steps below. This way you will be forced to answer the above questions even before you start playingwith Tapenade. These steps will also vastly ease your life when it comes to validate the differentiated programs. Finally, they willmake our life easier if we are to help you, and if we have to debug Tapenade!

1. First thing is to decide which is the part of the code that defines the function you want to differentiate. It is certainly not yourcomplete program! It must be a connex piece of code (call it the inside) with a unique beginning (call it the entry) and a uniqueend (call it the exit). This piece of code can very well contain calls to other procedures. Every run of your code that goesthrough the entry must then reach the exit. In particular, there must be no control jumps from outside to the inside, except tothe entry. Similarly, there must be no control jumps from the inside to the outside, except from the exit. Make this "inside"piece of code a subroutine or function (in FORTRAN) or a procedure (in C). You original code is therefore modified: youhave created a new procedure, and now there is a call to this procedure in place of the original piece of code to differentiate. Itnow looks like (FORTRAN style):

... whatever code is before ... C Beginning of the function I want to differentiate CALL MYFUNC(x1, x2,..., y1, y2,...)C End of the function I want to differentiate ... whatever code comes after ...

2. It goes without saying, but make sure you use some mechanism to make necessary variables travel to and from the newprocedure. You may use parameter passing, or FORTRAN common, or globals... as you wish. We shall call these variablesthe "parameters" of the procedure (even if they are indeed globals). It is very useful that you know these variables well, whatthey mean, what they are needed too, are they just used or just written and returned, or both, etc... Now you see why thispreparation work must be done by someone who knows the code!

3. Apply Tapenade in the "no­differentiation" mode, which is command­line option "­p". This will trigger several analysis but willgenerate a source code that should be equivalent to the original code, without any differentiation. The interest is to check thatTapenade is able to parse your source files correctly. Another interest is to see the warning messages issued by Tapenade.Some of these messages may indicate situations where things can go wrong. Therefore, we advise you to type somethingalong the lines of:

$> tapenade ­p ­o nodifferentiated "source files but no include files"

This will read all your source files (do not provide the include files in the command line because Tapenade will read themautomatically when needed). Your source code will be parsed and analyzed, and this will result in a filenodifferentiated_p.msg. It contains a (verbous) list of all warning and error messages, that you should browsethrough, with the help of the documentation on messages. The explanations in the reference manual will help you find whichmessages can be discarded (e.g., very often, the implicit conversions between REAL*8 and DOUBLE PRECISION), andwhich ones can't. Tapenade should have generated by now a new source file nodifferentiated_p.[f/f90/c], thatcontains a duplicate of your original source, with "_P" appended to procedure names. You may look at it to check that it isidentical to your source, and you may call it instead of the original code and check you get the same results. Your code couldnow look like:

... whatever code is before ... C Beginning of the function I want to differentiate CALL MYFUNC_P(x1, x2,..., y1, y2,...)C End of the function I want to differentiate ... whatever code comes after ...

THE FOLLOWING PREPARATION STEPS WERE USEFUL

FOR THE OLD VALIDATION METHOD ONLY

We keep them here for the record only, but they are not necessary!

1. Write a procedure that saves all these "parameters" of MYFUNC into some storage (can be static arrays), and a procedure thatrestores the saved values into these parameters. You may test them inside your modified code, just to make sure! Your codecould now look like:

... whatever code is before ... CALL SAVE_NECESSARY_PARAMS(x1, x2,..., y1, y2,...) CALL RESTORE_NECESSARY_PARAMS(x1, x2,..., y1, y2,...)C Beginning of the function I want to differentiate CALL MYFUNC(x1, x2,..., y1, y2,...)C End of the function I want to differentiate ... whatever code comes after ...

2. Call the new procedure twice in a row, inside the same execution. check you obtain the same results. So you will be surethat there is no unwanted side­effect or undefined­variable­used in your code to be differentiated. Your code could now looklike:

... whatever code is before ... C Save what is necessary in the current state CALL SAVE_NECESSARY_PARAMS(x1, x2,..., y1, y2,...)C Beginning of the function I want to differentiate CALL MYFUNC(x1, x2,..., y1, y2,...) res1_y1 = y1 res1_y2 = y2 ... C Prepare for the second run CALL RESTORE_NECESSARY_PARAMS(x1, x2,..., y1, y2,...)C Call MYFUNC again CALL MYFUNC(x1, x2,..., y1, y2,...) res2_y1 = y1 res2_y2 = y2 ... make sure all res1_yyy == res2_yyy C End of the function I want to differentiate ... whatever code comes after ...

3. Now choose which ones among the above "parameters" you want to differentiate (the "dependents"), with respect to whichones (the "independents"). Beware these must all be elements of the set of "parameters". We do not differentiate a procedurewith respect to its local variables, constants, or with respect to variables that the procedure does not see. In particular, ithappens that some of these parameters are dynamically allocated (resp. freed). Then it is essential that the allocation (resp.free) is outside of the procedure to be differentiated. Although this is not as essential, it is also good politics to avoid dynamicallocation/freeing anywhere inside the procedures to be differentiated. This may cause hard problems especially with thereverse mode.

4. Why not try and compute a (approached) derivative by divided differences then? It's maybe a little early but you will need todo it anyway for the old validation step. Define a differentiated variable for each of the dependent and independent. Initializethem all to 1.0. Modify each independent inputs by "epsilon" times its derivative variable. Run the procedure once and keepthe result. Reset the inputs, this time without the "epsilon" modification, and run again. Then evaluate the difference ofoutputs, divided by "epsilon". It is the sum of all columns of the Jacobian. For instance, if you want all inputs to beindependent and all outputs to be dependent, and setting "ddeps" the epsilon, your code could now look like:

... whatever code is before ... C Initialize the derivatives of the independent and dependent parametersC (don't forget to declare these variables !): x1d = 1.0 x2d = 1.0 ... y1d = 0.0 y2d = 0.0 ... C Save what is necessary in the current state CALL SAVE_NECESSARY_PARAMS(x1, x2,..., y1, y2,...)C Modify the current state: ddeps = 1.d­5 x1 = x1 + ddeps*x1d x2 = x2 + ddeps*x2d ... C Call F(X + epsilon*Xd) CALL MYFUNC(x1, x2,..., y1, y2,...) res1_y1 = y1 res1_y2 = y2 ... C Prepare for the second run CALL RESTORE_NECESSARY_PARAMS(x1, x2,..., y1, y2,...)C Call F(X) CALL MYFUNC(x1, x2,..., y1, y2,...) res2_y1 = y1 res2_y2 = y2 ... C Compute approx_Yd = (F(X+epsilon*Xd) ­ F(X))/epsilon

approx_y1d = (res1_y1 ­ res2_y1)/ddeps ... make sure these derivatives make sense! C End of the function I want to differentiate ... whatever code comes after ...

Independent and Dependent sets. Initialization and usage of the differentiated parameters What variables should I list in the "­vars" and "­outvars" arguments of the differentiation command? (also known as theindependent and dependent parameters)Also related: for my root procedure, the differentiated procedure that I obtain has additional differentiated parameters.What value should I give to these variables upon calling the differentiated root procedure?What value will I obtain in these variables upon return?

To begin with, what we refer to as "parameters" here is not limited to formal parameters in the procedure's argument list. The following applies justas well to global parameters such as those in a COMMON or in the variables of a USE'd module. If these variables are active, differentiated variableswill appear close to them, and if these active variables are inputs or outputs, you will have to think about providing their initial value and retrievingtheir result value.

Now some notation. Suppose we are calling Tapenade on some root procedure F, and call V some parameter of F. Although we advocatedifferentiating in a single batch all the subroutines that define the mathematical function of interest, it may help in the following explanation to thinkof F as only a part of this big mathematical function. Then in general there may be some upstream code U that runs before F is called, and somedownstream code D that runs after F is called. The complete mathematical function is implemented by U; F; D. This code sequence has aninput that we call the "main entry", and an output that we call the "main result". The general goal is to compute the derivatives of the main result withrespect to the main entry. Assuming that U and D are also differentiated in some way, the questions are: should V go into the independent anddependent sets, what is the new parameter Vd (or Vb) of the differentiated F_D (or F_B), what value should be provided into it, and what value willit contain upon exit? The header of each differentiated root procedure answers some of these questions. However, some explanations may beuseful.

Question 1: What should the independent and dependent parameters be set as?

It is wise not to think of those as the sets of inputs and outputs of F that are connected differentiably through F. Tapenade can find thisby itself !What Tapenade does expect from you is information about the context outside F. As a rule of thumb:

Think of the independents as the inputs of F that are influenced by the "main entry", i.e. that are differentiably connected to the mainentry through U. In the degenerate case where there is no U (empty U), the independents must be the inputs of F for which you, theend­user, promise to give a tangent derivative or expect a reverse derivative.Think of the dependents as the outputs of F that influence the "main result", i.e. that are differentiably connected to the main resultthrough D. In the degenerate case where there is no D (empty D), the dependents must be the outputs of F for which you, the end­user, expect a tangent derivative or promise to give a reverse derivative.

For example, if the context outside F is U(in:a,out:b); F(in:b,out:c); D(in:b,in:c,out:d);b is used differentiably later in D. Therefore, even if F only reads b and does not modify it, b must be in the set of dependents.Another possible mistake comes with loops. If the code looks like

U(in:a,out:b); loop F(in:b,out:c); D(in:c,out:d);the downstream code after F includes F itself, because of the loop, and therefore b must be in the dependents.

Please notice that Tapenade may remove variables from the independents or dependents sets that you provided. Tapenade detects superfluousindependents, i.e. inputs that have no differentiable influence on any dependent. Similarly a superfluous dependent is an output that doesn'tdepend differentiably on any independent and isn't independent itself. Superfluous variables are removed from their set, and a "command"message is issued. If you don't specify any independents (resp. dependents), Tapenade chooses for you the largest possible set, using itsdependence analysis to remove superfluous elements.

There are also situations where Tapenade adds variables to the independent or dependent sets that you provided for a root procedure. In tangentmode, if V is in the independent set and Vd may be overwritten in F_D, then V is added into the dependent set. Conversely in reverse mode, if V isin the dependent set and Vb may be overwritten in F_B, then V is added into the independent set. In both cases a "command" message is issued.This is Tapenade's way of warning the end­user that, although V was not in the dependent (resp. independent) set provided, the original Vd (resp.Vb) provided will be modified by the derivative procedure, and therefore will loose its original value and should be used with care after that.

Question 2: What is the effect of a variable being in the independent and/or dependent sets, on the requiredinitialization and returned value of its derivative?

Whatever independents and dependents sets you provided, it is wise to check these sets modified and actually used by Tapenade, as listed in thecomment header of the differentiated procedure. These sets determine how the derivatives must be initialized and what they will contain upon exit.This information is also summarized in the comment header of every differentiated root procedure.

The following table shows and explains the possible cases. In tangent mode, things are straightforward. In reverse mode things prove a little moresurprising, but they are "logical" consequences of the transposition of the Jacobian matrix that is behind the reverse mode. To give a rough idea, inreverse mode the flow of data goes in the reverse direction: If V is an output of F, then F_B must be called with an additional input VB, andconversely if V is an input of F, then F_B will return an additional output VB. When a variable is purely read by F, its derivative is purelyincremented (and vice versa)... oh yes, you're right it's complicated! But we never said that the reverse mode is simple, it's only extremely efficient

and worth the effort!

If V is listed as ... then in tangent mode, Vd... then in reverse mode, Vb...

... an independent(i.e. "varying")input only

... is a required input, and an unspecified output. Upon entry,F_D expects a value in Vd, which should be in general thederivative of V upon F entry with respect to the "main entry".In the particular case where this derivative is null, it is theresponsibility of the calling context to set Vd to 0.0 beforethe call to F_D.

Depending of what F does to V, Vd may very well bemodified by F_D: since V is not in the dependent outputs,the value of Vd upon F_D exit has no meaning and shouldnot be used. This is ok if V upon F exit has no influence onthe "main result".

When F is a differentiation root procedure, this appears inthe RW status in the header of F_D as "V:in" or "V:in­killed" to stress when Vd is modified by F_D.

... is a result only. Upon F_B exit, Vb contains the derivative ofthe "main result" with respect to the V upon F entry. This ispossible because F_B expects as input the derivative of the"main result" with respect to each of F's dependent outputs.

Since V is not in the dependent outputs, F_B expects nomeaningful value provided at call time in Vb. Actually,whatever is in there will be happily discarded, reset to zero,and overwritten. Therefore its value upon F_B entry will belost.

When F is a differentiation root procedure, this appears in theRW status in the header of F_B as "V:out" or "V:zero"when the Vb is null upon return from F_B.

... a dependent (i.e."useful") outputonly

... is a result only. Upon F_D exit, Vd contains the derivativeof V upon F exit with respect to the "main entry". This ispossible because F_D expects as input the derivatives ofF's independent inputs with respect to the "main entry".

Since V is not in the independent inputs, the value of Vdupon F entry is meaningless. Actually, whatever is in therewill be happily discarded and overwritten, Therefore itsvalue upon F_D entry will be lost.

When F is a differentiation root procedure, this appears inthe RW status in the header of F_D as "V:out" or"V:zero" when the exit Vd is null.

... is a required input, and an unspecified output. Upon entry,F_B expects a value in Vb, which should be in general thederivative of the "main result" with respect to V upon F exit. Inthe particular case where this derivative is null, it is theresponsibility of the calling context to set Vb to 0.0 before thecall to F_B.

Depending of what F does to V, Vb may very well be modifiedby F_B: since V is not in the independent inputs, the value ofVb upon F_B exit has no meaning and should not be used.This is ok if V upon F entry does not depend on the "mainentry".

When F is a differentiation root procedure, this appears in theRW status in the header of F_B as "V:in" or "V:in­killed" to stress when Vb is modified by F_B.

... independent anddependent

... is a required input and a result. Upon entry, F_D expects avalue in Vd, which should be in general the derivative of Vupon F entry with respect to the "main entry". In theparticular case where this derivative is null, it is theresponsibility of the calling context to set Vd to 0.0 beforethe call to F_D.

Upon F_D exit, Vd contains the derivative of V upon F exitwith respect to the "main entry". This is possible becauseF_D expects as input the derivatives of F's independentinputs with respect to the "main entry".

When F is a differentiation root procedure, this issummarized in the RW status of V in the header of F_D. Thisstatus can start with "in" when the entry value of Vd is used,and continues with

"out" when Vd is modified by F_D"zero" when Vd is reset to zero by F_Dor nothing when Vd is not modified by F_D

... is a required input and a result. Upon entry, F_B expects avalue in Vb, which should be in general the derivative of the"main result" with respect to V upon F exit. In the particularcase where this derivative is null, it is the responsibility of thecalling context to set Vb to 0.0 before the call to F_B.

Upon F_B exit, Vb contains the derivative of the "main result"with respect to the V upon F entry. This is possible becauseF_B expects as input the derivative of the "main result" withrespect to each of F's dependent outputs.

When F is a differentiation root procedure, this is summarizedin the RW status of V in the header of F_B. This status canstart with "in" when the entry value of Vb is used, andcontinues with

"out" when Vb is modified by F_B"zero" when Vb is reset to zero by F_Bor nothing when Vb is not modified by F_B

The RW status can also be "incr", meaning that Vb is simplyincremented and not otherwise read nor written by F_B.

Example: consider for instance procedure F below, that should illustrate many of the possible situations.

subroutine F(u2,v1,v3,w2) real u1,u2,u3,v1,v2,v3,w1,w2,w3 common /cc/u1,u3,v2,w1,w3c u3 = u3+v1*u1 u3 = u3­u2*v2 u2 = 3.0*u1*v1 u3 = u3+G(u2)c v3 = v3+2.0*v1 v3 = v3­v2 v2 = 3.0*v1 v3 = v3+G(v2)c w3 = w3+v1*v2 w3 = w3­w2*u1

w2 = 3.0*w1 w3 = w3+G(w2) end

real function G(x) real x G = x*x end

Function G is some pure user­defined function. We built this code so that all *1 parameters are only read by F, *2 parameters are read, thenoverwritten and maybe read again, and *3 parameters are only incremented, which is an interesting case for reverse differentiation. Parametersare scattered as formal arguments or global variables, just to show it doesn't matter.

Suppose that the end­user decided that all v* and w* parameters are independents, and all u* and w* are dependents, e.g. using the command­line option:

­head "F(v1 v2 v3 w1 w2 w3)>(u1 u2 u3 w1 w2 w3)"

In tangent mode, procedure F_D is In reverse mode, procedure F_B is

C Differentiation of f in forward (tangent) mode:C variations of useful results: u3 v2 w3 w2 u2C with respect to varying inputs: v2 w1 w3 v1 w2C RW status of diff variables: u3:out v2:in­outC w1:in w3:in­out v1:in w2:in­out u2:out SUBROUTINE F_D(u2, u2d, v1, v1d, v3, w2, w2d) IMPLICIT NONEC REAL u1, u2, u3, v1, v2, v3, w1, w2, w3 REAL u2d, u3d, v1d, v2d, w1d, w2d, w3d COMMON /cc/ u1, u3, v2, w1, w3 REAL G REAL G_D REAL result1 REAL result1d COMMON /cc_d/ u3d, v2d, w1d, w3dC u3d = u1*v1d u3 = u3 + v1*u1 u3d = u3d ­ u2*v2d u3 = u3 ­ u2*v2 u2d = 3.0*u1*v1d u2 = 3.0*u1*v1 result1d = G_D(u2, u2d, result1) u3d = u3d + result1d u3 = u3 + result1C v3 = v3 + 2.0*v1 v3 = v3 ­ v2 v2d = 3.0*v1d v2 = 3.0*v1 result1 = G(v2) v3 = v3 + result1C w3d = w3d + v1d*v2 + v1*v2d w3 = w3 + v1*v2 w3d = w3d ­ u1*w2d w3 = w3 ­ w2*u1 w2d = 3.0*w1d w2 = 3.0*w1 result1d = G_D(w2, w2d, result1) w3d = w3d + result1d w3 = w3 + result1 END

C Differentiation of f in reverse (adjoint) mode:C gradient of useful results: u3 w1 w3 w2 u2C with respect to varying inputs: u3 v2 w1 w3 v1C w2 u2C RW status of diff variables: u3:in­out v2:outC w1:incr w3:in­out v1:out w2:in­out u2:in­out SUBROUTINE F_B(u2, u2b, v1, v1b, v3, w2, w2b) IMPLICIT NONEC REAL u1, u2, u3, v1, v2, v3, w1, w2, w3 REAL u2b, u3b, v1b, v2b, w1b, w2b, w3b COMMON /cc/ u1, u3, v2, w1, w3 REAL G REAL result1 REAL result1b COMMON /cc_b/ u3b, v2b, w1b, w3b CALL PUSHREAL4(u2)C u2 = 3.0*u1*v1 CALL PUSHREAL4(v2)C v2 = 3.0*v1C w2 = 3.0*w1 result1b = w3b CALL G_B(w2, w2b, result1b) w1b = w1b + 3.0*w2b w2b = ­(u1*w3b) v1b = v2*w3b v2b = v1*w3b CALL POPREAL4(v2) result1b = u3b CALL G_B(u2, u2b, result1b) v1b = v1b + u1*3.0*u2b + u1*u3b + 3.0*v2b CALL POPREAL4(u2) u2b = ­(v2*u3b) v2b = ­(u2*u3b) END

You can see that Tapenade took u1 and w1 out of the set of dependentsbecause they are not influenced differentiably by any independent. Italso took v3 out of the independents because it has no differentiableinfluence on any dependent and is not dependent itself. Also, Tapenadeadded v2 to the set of dependents because its derivative has to bemodified by F_D.

Since v1, v2, w1, w2, and w3 are independent (i.e. in the comment"varying inputs"), procedure F_D expects to be provided with theirderivatives v1d, v2d, w1d, w2d, and w3d with respect to the "mainentry". After modification of the independent and dependent sets byTapenade, there is simply no u1d nor v3d since they are neitherindependent nor dependent. The value upon F_D entry of the otherderivative variables u2d and u3d are meaningless and are happily

You can see that Tapenade took u1 out of the set of dependentsbecause it is not influenced differentiably by any independent and it isnot an independent itself. It also took v3 out of the independentsbecause it has no differentiable influence on any dependent and is notdependent itself. Also, Tapenade added u2 and u3 to the set ofindependents because their derivative has to be modified by F_B.Actually this is not the case for u3 because u3 is only incremented in F,but Tapenade does not detect this (yet?).

Since u2, u3, w1, w2, and w3 are dependent (i.e. in the comment"useful results"), procedure F_B expects to be provided with thederivatives u2b, u3b, w1b, w2b, and w3b of the "main result" withrespect to them. After modification of the independent and dependentsets by Tapenade, there is simply no u1b nor v3b since they are neitherindependent nor dependent. The input values of the other derivatives

overwritten inside F_D. So beware that whatever was in there beforecalling F_D may be lost.

The dependent are variables u2, u3, v2, w2, and w3 (i.e. in thecomment "useful results"), so that they contain upon F_D exit thederivative of the corresponding original variable upon F_D exit withrespect to the "main entry". Variables u1, v1, v3, and w1 are notdependent, therefore the exit value of their derivative is not specified. Infact these derivatives are not overwritten so there is no risk.

In other words, variables u1, v1, v3, and w1 are only read inside Fand therefore their derivatives either don't exist or are only read by F_D.They retain their input value.

v1b and v2b are meaningless and may be happily overwritten insideF_B. So beware that whatever was in there before calling F_B may belost.

At the other end of F, the independent are variables u2, u3, v1,v2, w1, w2, and w3 ("varying inputs"). Therefore they containupon exit from F_B the derivative of the "main result" with respect totheir corresponding original variable, upon F entry.

Notice furthermore that variables u3 and w3 are only incremented insideF and therefore their derivatives u3b and w3b are only read by F_B.They retain their input value. Still, Tapenade was not able to detect thatand returned the more general RW status "in­out"

Validation of the differentiated programs Help, the differentiated program crashes!How can I be sure that the differentiated program really computes what I want?How can I check the derivatives and find the place where the differentiated program goes wrong?

This is a difficult issue. Differentiated programs are very long and complex and you may find it hard to check them by hand, even ifwe try to keep them readable. Here is a proposed method to validate/debug the code produced by Tapenade. It uses Tapenadeoptions that are available only in the command­line version (local installation). It also uses two files that are in theADFirstAidKid directory of your local installed Tapenade files. If you don't have a locally installed Tapenade, please followthese steps. Also, to ease our validation mechanism, we strongly advise you to follow a few preparation steps, before you evenstart to differentiate anything. Validation is by no means easy. Although we try to automate the process, it still requires goodknowledge of the principles of AD, especially for the adjoint mode. The sad thing with Tapenade's validation mechanism is that youmay run into bugs of Tapenade itself (otherwise you wouldn't be reading this section anyway!) but you may also run into bugs of thevalidation mechanism itself.

In the following, words between double quotes and drawn in this color (such as "SHAPE") are indeed templates, that must beinstantiated by actual strings depending on the context. Square brackets represent additional arguments, and the star representsrepetition.In the example pieces of programs, actual pieces of code are written in this sort of dark red, short strings in double quotes are"patterns" as above, and

"entiere lines between double quotes and drawn in the pattern color"

describe a piece of code that you must write at that place.

The choice of Independent and Dependent parameters may interfere with the validation process. This is explained at the end of thissection, but we'd rather warn you right now. Make sure you read this part if validation fails close to the entry­into or exit­from yourcode.

CHECK THE SIZES OF PRIMITIVE TYPES:

If the program crashes, there is a small chance that this comes from the sizes of basic types on your system. To test these sizes, gointo the ADFirstAidKid directory, then:

1) adapt the compilation commands $(FC) and $(CC) in the Makefile2) make testMemSize3) execute testMemSize4) look at the sizes printed.

The standard Tapenade expects: integers:4 ; integers*8:8 ; reals:4 ; reals*8:8 ; doubles:8 ; complexes:8 ; complexes*8:8 ;complexes*16:16 ; double complexes:16 ; logicals:4 ; characters:1if testMemSize on your system gives different sizes, use the type size options that change the default sizes to what your systemexpects.

CHECK THE TANGENT DERIVATIVES WITH RESPECT TO DIVIDED DIFFERENCES:

Comparison with divided differences is the standard way to test the tangent derivatives. It amounts to running your originalsubroutine P with a perturbed input X+*Xd, then with X, then computing:

You should get nearly the same result as when running the tangent differentiated subroutine P_d(X,Xd,Y) With a "good" epsilon,differences should be only on a few last decimals. There is a lot to say on the choice of a good epsilon. It depends on the algorithm,the data ... and maybe also the phase of the moon! Also, it is certainly more accurate to compute the centered divided differences,but we generally don't need this degree of accuracy for our validation. To run the test, start from the prepared code according to therecommended code preparation. Suppose the "central" part looks like:

... whatever code is before ... C Beginning of the procedure to differentiate CALL MYFUNC(x1, x2,..., y1, y2,...)C End of the procedure to differentiate

... whatever code comes after ...

Define two copies of your complete code, that will differ only in this "central" part.

The first copy, that we will call "codeDD1", compiles into an executable that we will call "exeDD1". Instead of callingMYFUNC, it calls its tangent differentiated version MYFUNC_D, calling a few debugging primitives before and after. The centralpart of "codeDD1" is thus:

... whatever code is before ...

Now initialize the input differentiated variables, to define the chosen differentiation direction, e.g.: x1d = 1.5d0 x2d = (/­1.d0,2.3d0,­0.5d0,.../) ... Now prepare for the test: CALL DEBUG_TGT_INIT1(1.d­8, 1.d­5, .1d0) CALL DEBUG_TGT_INITREAL8(x1, x1d) CALL DEBUG_TGT_INITREAL8ARRAY(x2, x2d, ns) ... CALL DEBUG_TGT_CALL('MYFUNC', .true., .false.)C Beginning of differentiated part CALL MYFUNC_D(x1, x1d, x2, x2d, ..., y1, y1d, y2, y2d, ...)C End of differentiated part Now terminate the test: CALL DEBUG_TGT_EXIT() CALL DEBUG_TGT_CONCLUDEREAL8('y1', y1, y1d) CALL DEBUG_TGT_CONCLUDEREAL8ARRAY('y2', y2, y2d, 20) ...

... whatever code comes after ...

The second copy, that we will call "codeDD2", compiles into an executable that we will call "exeDD2". It is strictly identicalto "codeDD1" except that it calls DEBUG_TGT_INIT2 instead of DEBUG_TGT_INIT1

A few remarks and explanations:

Each copy "codeDD1" and "codeDD2" will be executed independently. Each must use or read from the same data files asthe original code, and should behave exactly the same until reaching the central part. You don't need to duplicate everysource file to define these copies "codeDD1" and "codeDD2". You probably need only to duplicate the file that containsthe "central" part. The rest should be taken care of by the compilation and link commands.The code for MYFUNC_D must be built by Tapenade by tangent differentiation of the source code, with the only additionalcommand­line option ­debugTGT. The same differentiated code is used by "exeDD1" and "exeDD2".Each executable "exeDD1" and "exeDD2" must be linked with the debugging library provided in the "First­Aid Kit". Thislibrary consists of file debugAD.f and its include file debugAD.incThe calls to DEBUG_TGT_INIT1 or to DEBUG_TGT_INIT2 must have the same arguments. The first is the "epsilon" usedfor divided differences. The second is called "almost­zero" (see below). The third is the "threshold" (see below) errorpercentage above which a message is printed.In the prepared code above, notice that there is a statement that initializes each input differentiated variable. You are free tochoose the values, therefore choosing the direction of your tangent. Naturally, arrays require several values.A few lines later, before calling MYFUNC_D, you must insert one initialization call for each input differentiated variable. Forarrays, initialization procedure DEBUG_TGT_INITREAL8ARRAY takes an extra argument, which is the size of the array. Forsingle precision REAL's, replace REAL8 by REAL4.The same remarks apply to the DEBUG_TGT_CONCLUDE... debug termination calls.Don't forget to declare all differentiated variables in the declaration part of your codes "codeDD1" and "codeDD2".

When "exeDD1" and "exeDD2" are ready, run the divided differences test by just typing the command line:

$> exeDD1 | exeDD2

using a UNIX pipe "|" (or equivalent). What happens behind the scene is: exeDD1 runs on the input "plus epsilon times direction".At selected moments ("points" in execution), it sends the value of selected variables into std­out, therefore into the pipe. In parallel, exeDD2 runs on the input "without epsilon". At the same selected points, it reads the value of the same selectedvariables from std­in, therefore from the pipe. Using its own value of the same variable, exeDD2 computes de derivativeapproximation by divided differences (the "DD derivative") and compares it with the analytical derivative. If the DD derivative isalmost zero (i.e. absolute value below the given "almost­zero"), the comparison is not done because Tapenade code sometimesdoesn't reinitialize zero derivatives (to save time). Otherwise, if the percentage of difference is more that the "threshold", a messageis printed on the screen. What you obtain on the screen looks like what follows. In this example, we introduced an error into thetangent differentiated code, inside the tangent derivative of some function called "POLYSURF":

$> exeDD1 | exeDD2Starting TGT test, epsilon=0.1E­07, zero=0.1E­04, errmax=10.0%=========================================================== AT:entry OF MYFUNC AT:entry OF POLYPERIM AT:middle OF POLYPERIM AT:entry OF INCRSQRT AT:exit OF INCRSQRT AT:entry OF INCRSQRT

AT:exit OF INCRSQRT AT:exit OF POLYPERIM AT:entry OF POLYPERIM AT:middle OF POLYPERIM ... lots of lines deleted here ... AT:exit OF POLYPERIM AT:middle OF MYFUNC AT:entry OF POLYSURF AT:middle OF POLYSURFpp: ­0.1000000000000000E+01 (ad) 90.0% DIFF WITH (dd) ­0.1000000011686097E+02 AT:entry OF INCRSQRTpp: 0.3001997501343353E+01 (ad)150.0% DIFF WITH (dd) ­0.5998002627904953E+01 AT:exit OF INCRSQRTpolysurf: 0.3001997501343353E+01 (ad)150.0% DIFF WITH (dd) ­0.5998002627904953E+01 AT:exit OF POLYSURFy2: 0.3787447771453199E+13 (ad) 92.3% DIFF WITH (dd) 0.2901041015625000E+12 AT:exit OF MYFUNCFinal result y2: 0.3787447771453199E+13 (ad) 92.34038% DIFF WITH (dd) 0.2901041015625000E+12===========================================================

From this listing, you can deduce that DD and AD derivatives start to diverge between the "middle" point of subroutine POLYSURFand the following call to subroutine INCRSQRT, and the faulty derivative is that of variable pp. The error in the differentiated code isprobably nearby. Now you can start to inspect the code from that point, identify the bug, and maybe send us a bug report !

It is clear what the "entry" and "exit" debug points of a procedure are. The "middle" debug point is placed automatically byTapenade debug mode. You can easily find it in the differentiated code. For more accuracy, you can add debug points by placing$AD DEBUG­HERE directives in your original source. For instance:

IF (pp.EQ.0) pp=nsC$AD DEBUG­HERE CP4 cp.eq.4 dx = X(cp)­X(pp)

defines a new debug point named "CP4". The second argument (optional, default is true) of the directive is a test (with no whitespace in it, please), and debug point is active only when the test is true. If no test is given, the debug point is always active. Therecan be a third argument (optional, default is false) that, when true, forces the debug point to be active even if debug of the enclosingprocedure is not active (see below).

By default, debugging occurs on every "active" variable at every debug point of every subroutine called. You can control debuggingfor the complete call tree below a particular subroutine call, by placing the $AD DEBUG­CALL directive in your original sourcebefore this particular call. For instance:

C$AD DEBUG­CALL false perim = perim+ i*POLYPERIM(X,Y,ns)

turns debugging off for the call tree below this function call. The false can be replaced by something more elaborate (no whitespace in the test please!). When debugging is turned off inside a particular sub tree of the call tree, no test is made and no messageis printed for this sub call tree. However, you have the possibility to force debugging inside this sub call tree by adding a thirdargument to some $AD DEBUG­HERE or $AD DEBUG­CALL inside. This third argument must be a test. When it evaluates to true,debugging is temporarily turned on again for the debug point or inside the call in question.

CHECK THE REVERSE DERIVATIVES USING THE DOT­PRODUCT TEST:

The dot­product test relies on the fact that for every program, procedure, or part P of a procedure, for every vector Xd in the space ofthe inputs of P and for every vector Yb in the space of the outputs of P, the following scalar:

Yb* . JP . Xd(where * is transposition and JP is the Jacobian (derivative matrix) of the function implemented by P) can be computed from left toright or from right to left:

(Yb* . JP) . Xd = Yb* . (JP . Xd)Since (Yb* . JP) is in fact Xb*, the output of the reverse differentiated program P_B applied to Yb, and conversely (JP . Xd)is in fact Yd, the output of the tangent differentiated program P_D applied to Xd, we find that correct tangent­ and reverse­differentiated codes P_D and P_B must be such that:

Xb* . Xd = Yb* . Ydor in terms of dot­product

(Xb . Xd) = (Yb . Yd)

We assume that the tangent­differentiated code is correct, for instance because it was validated by the divided­differences method.We propose a method to validate the reverse differentiated program P_B by checking that this dot­product equality holds and if itdoesn't, to locate the part of the reverse differentiated program where the equality doesn't hold.Suppose again that the "central" part looks like:

... whatever code is before ...

C Beginning of the procedure to differentiate CALL MYFUNC(x1, x2,..., y1, y2,...)C End of the procedure to differentiate ... whatever code comes after ...

Define again two copies of your complete code, that will differ only in this "central" part.

The first copy, that we will call "codeBB1", compiles into an executable that we will call "exeBB1". Instead of callingMYFUNC, it calls its reverse differentiated version MYFUNC_B, calling a few debugging primitives before and after. The centralpart of "codeBB1" is thus:

... whatever code is before ...

Now initialize the REVERSE differentiated variables, e.g.: (this step is NOT absolutely necessary for debug !) x1b = 0.d0 DO i=1,ns x2b(i) = 0.d0 ENDDO y1b = 1.d0 DO i=1,20 y2b(i) = 1.d0 ENDDO ... Now prepare for the test: CALL DEBUG_BWD_INIT(1.d­5, 0.1d0, 0.87d0) CALL DEBUG_BWD_CALL('MYFUNC', .true.)C Beginning of differentiated part CALL MYFUNC_B(x1, x1b, x2, x2b, ..., y1, y1b, y2, y2b, ...)C End of differentiated part Now terminate the test: CALL DEBUG_BWD_EXIT() CALL DEBUG_BWD_CONCLUDE()

... whatever code comes after ...

The second copy, that we will call "codeBB2", compiles into an executable that we will call "exeBB2". Again, only itscentral part is modified into the following code:

... whatever code is before ...

Now initialize the TANGENT differentiated variables, e.g.: (this step is NOT absolutely necessary for debug !) x1d = 1.d0 DO i=1,ns x2d(i) = 1.d0 ENDDO y1d = 0.d0 DO i=1,20 y2d(i) = 0.d0 ENDDO ... Now prepare for the test: CALL DEBUG_FWD_INIT(1.d­5, 0.1d0, 0.87d0) CALL DEBUG_FWD_CALL('MYFUNC')C Beginning of differentiated part call MYFUNC_D(x1, x1d, x2, x2d, ..., y1, y1d, y2, y2d, ...)C End of differentiated part Now terminate the test: CALL DEBUG_FWD_EXIT() CALL DEBUG_FWD_CONCLUDE()

... whatever code comes after ...

A few remarks and explanations:

Each copy "codeBB1" and "codeBB2" will be executed independently. Each must use or read from the same data files asthe original code, and should behave exactly the same until reaching the central part. You don't need to duplicate everysource file to define these copies "codeBB1" and "codeBB2". You probably need only to duplicate the file that containsthe "central" part. The rest should be taken care of by the compilation and link commands.Note that "codeBB1" makes use of the reverse­differentiated code MYFUNC_B and of the associated differentiated variablesx1b etc. The code for MYFUNC_B must be built by Tapenade by reverse differentiation of the source code, with the onlyadditional command­line option ­debugADJ.Conversely, "codeBB2" makes use of the tangent­differentiated code MYFUNC_D and of the associated differentiatedvariables x1d etc. The code for MYFUNC_D must be built by Tapenade by tangent differentiation of the source code, with theonly additional command­line option ­debugADJ.Each executable "exeDD1" and "exeDD2" must be linked with the debugging library provided in the "First­Aid Kit". Thislibrary consists of file debugAD.f and its include file debugAD.incThe calls to DEBUG_BWD_INIT or to DEBUG_FWD_INIT must have the same arguments. The first is called "almost­zero"(see below). The second is the "threshold" (see below) error percentage above which a message is printed. The third is the"seed" that influences the random choice of vectors Xd and Yb during testing.

In the prepared code above, notice that debug statements do not concern individual differentiated variables. This is because,unlike the divided­differences test, this dot­product test is global and cannot be applied to a single variable.Don't forget to declare all differentiated variables in the declaration part of your codes "codeBB1" and "codeBB2".

When "exeBB1" and "exeBB2" are ready, run the divided differences test by just typing the command line:

$> exeBB1 | exeBB2

using a UNIX pipe "|" (or equivalent). What happens behind the scene is: exeBB1 calls MYFUNC_B, which is conceptually split intoa sequence of code pieces between successive points. At the beginning of each (reverse differentiated) piece (actually it's what wecall its downstream end, because control flow is reversed...), pseudo­random values (using "seed") are assigned to the currentreverse differentiated variables Yb. At the end of the same piece (actually it's what we call its upstream end, because control flow isreversed...), the current reverse differentiated variables Xb are multiplied by dot­product with a randomly­filled Xd. The result is sentinto std­out, therefore into the pipe. In parallel, exeBB2 reads from std­in, therefore from the pipe, and reverses the sequence of incoming dot­products. exeBB2 thencalls MYFUNC_D, which is split in a way that matches the splitting of MYFUNC_B. At the beginning of each (tangent differentiated)piece, i.e. the upstream end of this piece, the same random values are assigned to the current tangent differentiated variables Xd.At the end of the same piece, i.e. the downstream end of this piece, the current tangent differentiated variables Yd are multiplied bydot­product with the same randomly­filled Yb. The dot­product is compared with the one coming from exeBB1. If the two dot­products are almost zero (i.e. absolute value below the given "almost­zero"), no comparison is done. Otherwise, if the percentage ofdifference is more that the "threshold", a message is printed on the screen. What you obtain on the screen looks like what follows.In this example, we introduced an error into the reverse differentiated code, inside the reverse derivative of some function called"POLYSURF":

$> exeBB1 | exeBB2Starting ADJ test, zero=0.1E­04, errmax=10.0%, random_incr=0.87E+00=========================================================== AT:entry OF MYFUNC AT:entry OF POLYPERIM AT:middle OF POLYPERIM ... lots of lines deleted here ... AT:exit OF POLYPERIM AT:middle OF MYFUNC AT:entry OF POLYSURF AT:middle OF POLYSURF49.0% DIFFERENCE!! fwd: 0.1156790000000001E+02 bwd: 0.2268820000000001E+02 AT:entry OF INCRSQRT AT:exit OF INCRSQRT AT:exit OF POLYSURF AT:exit OF MYFUNCEnd of ADJ test. 1 error(s) found. WARNING: testing alters derivatives!===========================================================

From this listing, you can deduce that there is an error in the reverse differentiation of the piece of code between the "middle" pointof subroutine POLYSURF and the following call to subroutine INCRSQRT. Now you can start to inspect the code from that point,identify the bug, and maybe send us a bug report !

As for the divided­differences test, the "entry" and "exit" debug points are clear enough, and the "middle" debug point isplaced arbitrarily by Tapenade. For more accuracy, you can add debug points by placing the same $AD DEBUG­HERE directivesin your original source. You can also place $AD DEBUG­CALL directives in your original source to control debugging more finelyinside the call tree. Notice however that the third argument of these directives is simply ignored by the dot­product test. In otherwords, if debugging is turned off for a given sub call tree, there is no way to turn it on locally inside the sub call tree. Notice finallythat it is delicate and dangerous to try and add new debug points by hand into the reverse differentiated code. On the other hand, itis rather easy and safe to turn off/on existing debug points inside the reverse differentiated code, without needing to re­differentiatewith different $AD DEBUG­**** directives. This may prove handy as running Tapenade on a big code can be long. To do so, youjust need to edit the reverse­differentiated code. In it you will find the debug points by their names.

To turn off/on testing inside the call tree below a particular call to procedure FOO, find this call to FOO_B, move up thedebugging if­then­else before it, to reach the call to

CALL DEBUG_BWD_CALL('FOO', "boolean­expression")you may change the boolean second argument at will.To turn off/on a debug point named "mypoint", find the test:

IF ("boolean­expression" .AND. DEBUG_BWD_HERE('mypoint')) THENand change the boolean before AND at will. Debug points named "entry", "exit", "beforeCall", and "afterCall" cannot and must not be turned off individually.

PROBLEMATIC INTERACTION WITH INDEPENDENT AND DEPENDENT PARAMETER SETS:

This problem was identified and studied with the help of Nicolas Huneeus, LSCE­IPSL, University of Versailles Saint­Quentin.Tapenade sometimes takes the liberty to modify the Independent and Dependent parameters provided by the end­user. Thereasons are described here and, although they are complex, we believe they are justified. This modification can be different intangent and adjoint modes.

This may cause problems in the tangent validation ("divided differences") because it affects the way derivatives areinitialized.

This will certainly cause problems in the adjoint validation ("dot product") if the tangent and adjoint codes are built with(in)dependent paramaters that don't match

As a result, the validation results may display unjustified "Differences", located at the entry or at the exit of the head procedure.

The sign you must look for is the occurrence of one of the messages:Command: Input variable(s) xxxx have no differentiable influence in pppp: removed fromindependentsCommand: Output variable(s) xxxx are not influenced differentiably in pppp: removed fromdependentsCommand: Input variable(s) xxxx have their derivative modified in pppp: added toindependentsCommand: Output variable(s) xxxx have their derivative modified in pppp: added to dependentswhen setting the message level high enough (command­line option ­msglevel 20).

There are two ways to turn around this problem:

either add the command­line option ­fixinterface and the Independent and Dependent parameters will not be modified,or modify by hand your Independent and Dependent parameter sets, according to the variables indicated by the abovemessages. Then the sets will be coherent again between the tangent and the adjoint codes.

Problems with web browsers (Macintosh): I'm using Internet Explorer for Macintosh, and Tapenade complains about syntax errors in my Fortran files. However I'm sure thesyntax is correct, and/or it works fine through the cut­and­paste interface.

This bug was found and fixed by David Pearson, University of Reading U.K. The reason might be that the Fortran files that you areuploading to us are sent as "application/x­macbinary" instead of "application/octet­stream". This can be fixed by modifying yourbrowser's preferences:

Open "Preferences", then click either on "File Helpers" if visible, or on "Receiving Files/File Helpers" sub­menu. You shouldget something like this.Click the "Add..." button.Enter a description, e.g. "Fortran files", and an extension, e.g. .f or .f90.Enter a MIME type application/octet­stream.Enter a File type. Four characters exactly are required here. Type e.g. FORT .Enter a File creator (as for "File type" above).Make sure the "Plain text" radio button is selected.Tick "Use for incoming" and "Use for outgoing" check­boxes.This should look like this. Then click "OK".

Other AD tools: What are the other Automatic Differentiation tools ? How do they compare to Tapenade ?

This is only our partial vision of the other AD tools. Maybe a better source is the www.autodiff.org site for the AutomaticDifferentiation community, managed by our colleagues in Aachen and Argonne.

There are AD tools that rely on program overloading rather than program transformation. In general this makes the tool easier toimplement. However some overloading­based AD tools can become very sophisticated and efficient, and represent a fair bit of hardwork too. Overloading­based AD tools exist only for target languages that permit some form of overloading, e.g. C++ and Fortran95.Overloading­based AD­tools are particularly adapted for differentiations that are mostly local to each statement, i.e. no fancy controlflow rescheduling is allowed. On the other hand, these local transformations can be very complex, more than what transformation­based AD tools generally provide. For instance, overloading­based AD­tools can generally compute not only first, but also second,third derivatives and so on, as well as Taylor expansions or interval arithmetic. Adol­C, from TU Dresden, is an excellent exampleof overloading­based AD tool. Also FADBAD/TADIFF

There are AD tools that transform the original source into a differentiated source. Tapenade is one of those. These tools share theirgeneral architecture, with a front­end very much like a compiler, followed with an analysis component, a differentiation component,and finally a back­end that regenerates the differentiated source. They differ in particular in the language that they recognize anddifferentiate, the AD modes that they provide, and some differences in AD strategies, mostly about the reverse mode. Otherstransformation­based AD tools are:

Adifor differentiates Fortran77 codes in tangent (direct) mode. Adifor once was extended towards the adjoint (reverse) mode(Adjfor), but we believe this know­how has now been reinjected into the OpenAD framework.Adic can be seen as the C equivalent of Adifor. However it is based on a completely different architecture, from the OpenADframework. This framework, very similar to Tapenade's architecture, claims that only front­end and back­end should dependon the particular language, whereas the analysis and differentiation part should work on a language­independent programrepresentation. Adic differentiates ANSI C programs in tangent mode, with the possibility to obtain second derivatives.OpenAD/F differentiates Fortran codes in tangent and adjoint modes. Its strategy to restore intermediate values in reverse ADis extremely close to Tapenade's. OpenAD/F is made of Adifor and Adjfor components integrated into the OpenADframework.TAMC, and its commercial offspring TAF differentiate Fortran files. Taf also differentiates Fortran95 files, under certainrestrictions. TAF is commercialized by the FastOpt company in Hamburg, Germany. TAF differentiates in tangent and reversemode, with the recompute­all approach to restore intermediate values in reverse AD. Checkpointing and an algorithm to avoiduseless recomputations (ERA) are used to avoid explosion of run­time. TAF also provides a mode that efficiently computesthe sparsity pattern of Jacobian matrices, using bitsets.

TAC++ is the C version of TAF. It is also developped by FastOpt. Presently, TAC++ only handles a large subset of C, and it isstill in its development phase, although making significant progress. Like TAF, it will provide tangent and adjoint modes, withthe same strategies, e.g. the recompute­all approach with checkpointing for the reverse mode.

There are AD tools that directly interface to an existing compiler. In fact, these are extensions to the compiler so that differentiatedcode is added at compile time. For instance the NAGWare Fortran95 compiler has AD facilities inside, that are triggered by userdirectives in the Fortran source. It so far provides tangent­mode differentiation.

There are AD tools that target higher­level languages, such as MATLAB. We know of ADiMat, MAD, and INTLAB. Even when theyrely on operator overloading, they may embed a fair bit of program analysis to produce efficient differentiated code.

Directory separator linux/windows: I need to use "\" as a separator in my paths, and Tapenade uses the UNIX "/" separator.

This happens when your files are stored in a WINDOWS (DOS) file system and you refer to them from a linux system such ascygwin. Use the "­parserfileseparator" option on the Tapenade command line to modify the path separator. For instance:

tapenade ... ­parserfileseparator "\" ...

or

tapenade ... ­parserfileseparator "/" ...

JAVA errors: NullPointerException, ArrayOutOfBoundsException, etc: What does it mean when I get System: java.lang.NullPointerExceptionas a message, or similarly an ArrayOutOfBoundsExceptionIt just means you ran into one of the remaining bugs in Tapenade. Please send us a bug report, and we will fix it.

Differentiating multiple top/head procedures, Differentiating Libraries: I want to differentiate many top/head proceduresI want to differentiate a top/headprocedure for different activity contextsI want to force the activity context of a given procedure to something larger than Tapenade's choiceI want to differentiate every entry of a library

The recent versions of Tapenade allow you to differentiate several top/head procedures at the same time. This is convenient fordifferentiating libraries. The advantage is the following: when two head procedures call the same internal procedure, the internalprocedure is differentiated only once, with an activity context which is the union of the contexts coming from each head procedure.In the older versions of Tapenade, where only one head procedure could be differentiated, users had to write a dummy topprocedure that called every entry of the library, arranging for the desired activity context for each call. This could prove tedious.

To avoid this tedious task, the syntax of the command line has changed as follows (the old syntax still works, though): The ­headoption now accepts a string between double quotes, in which you may specify any number of triplets:

Subroutine nameDependent outputsIndependent inputs

For instance

tapenade foo.f bar.f toto.f ... ­head "f1(y1 y3)/(x2) f2(b2 b4)/(a1 a2 a3)"

will differentiate head subroutine f1, differentiating outputs y1 and y3 with respect to x2 and at the same time head subroutine f2will be differentiated for dependent outputs b2 and b4 with respect to independent inputs a1, a2, and a3. You may split these headsubroutines into several ­head options. If you use this syntax, the options ­outvars and ­vars become useless. There is analternative syntax for people who want to say it the other way around, i.e. for instance

­head "f1(x2)>(y1 y3)"

In Fortran90, you may want to designate a subroutine or a variable name from a given module. You may do so using a % or a .character, equivalently. For instance:

­head "m5%f1(y1 y3 m2%v2%c4)/(x2)"

designates the procedure named f1 in module m5, and in the list of dependent outputs, you can see the component c4 of theglobal variable v2 from module m2. Of course we assume that v2 has public access and it is of a structured type with a field namedc4.

Finally, it happens that Tapenade finds that some differentiation head procedure actually has a differentiation context smaller thanthe one given by the user. Suppose for instance that you specify

­head "m5.f1(y1 y3 m2.v2)/(x2 x4)"

and actually no dependent output depends on x4, and at the same time y3 actually depends on no independent input. Tapenadewill "simplify" the command­line options, neglecting variables x4 and y3. This is ok in general, but annoying sometimes. Noticethat a message is displayed to explain the situation, but the differentiated subroutine will lack the differentiated arguments for x4and y3. The command­line option ­fixinterface will force the differentiated arguments to be present. Hopefully they areuseless, but they will be there. This is useful when feeding programs produced by Tapenade into optimization platforms that requirea fixed interface.

Download Tapenade Where can I download a local version of Tapenade?

You may download the latest version of Tapenade on your local system. First read our downloading policy, and register. You willget a link to a README.html file.

Old versions of Tapenade How can I get a previous version of Tapenade?

Well, to put it bluntly, you can't !

Sorry but keeping several versions available is a lot of work. More importantly, it contradicts our belief (hope, dream, ?) thatsuccessive versions of Tapenade only add new functionalities, improve existing ones, and fix bugs. In the vast majority of cases,this is justified.

However, it did happen in the past that switching to a new version posed a problem to a user. For instance when we introduced thePUSHCONTROL() in version 3.5, or the automatic modification of the user­provided independent and dependent sets in version2.2. In these cases, the problems could be solved by a small change in the end­user's environment, or a new command­line option.We think it was for the better.

So if you find yourself in this situation, we encourage you to analyze the problem and tell us which is the difference in thedifferentiated program that poses a problem. A small standalone source file would be great. Hopefully we can find a way so thatyou can use the new Tapenade and benefit from next improvements.

System errors of the "not found" sort: Something went wrong after installation: I get aException in thread "main" java.lang.NoClassDefFoundError: toplevel/Differentiatoror a messageTool: Parser not foundCheck your environment variable (e.g. shell variable) TAPENADE_HOME. This variable is used in file bin/tapenade (for the "LINUX" case) and in file bin/tapenade.bat (for the "WINDOWS" case). Also when switching to a new version of Tapenade, don't forget to update your TAPENADE_HOME variable to refer to the newversion, e.g.TAPENADE_HOME="install_dir"/tapenade3.5See README.html for more details.

System errors of the "out of memory" sort: I get aException in thread "main" java.lang.OutOfMemoryErrorThe Tapenade server complains my source file is too large!

You are probably differentiating a large program!

The Tapenade server will reject your files if the total size of files submitted is over 300000 characters, as many users can accessthe server simultaneously and too large files make is swap and slow down, or even crash. If you run into this limit, you mustprobably download a Tapenade executable on your local system. It is free for research non­commercial use.

By the way, it still happens that the server crashes because of too many users differentiating too large files. In this case the servershould restart automatically, so please wait for a moment and retry.

For a locally installed Tapenade, if the java heap overflows (java.lang.OutOfMemoryError), increase the heap size in thelaunching script file bin/tapenade (for the "LINUX" case) or in file bin/tapenade.bat (for the "WINDOWS" case).Set HEAP_SIZE=­mx2048m or even HEAP_SIZE=­mx4096m

Sending remarks, questions, and bug reports: Where should I submit questions about Tapenade ?How do I send bug reports to the Tapenade development team ?

First of all, this is a very good idea. Thanks !

Bug reports should be sent to the developers of Tapenade at this e­mail address: tapenade@lists­sop.inria.fr

On the other hand, questions and remarks should rather be sent to the tapenade­users mailing list. You will have to subscribe first, then you will be able to submit your questions and remarks by mailing to tapenade­users@lists­sop.inria.fr. Maybe someother user of Tapenade will answer even before we do! Naturally, frequently asked questions should eventually be included here.

If you really don't want to subscribe, then you may just send an e­mail to the developers of Tapenade

More specifically about bugs, there are some data that we need in order to track the bug efficiently. If the error you got contained theJava execution stack, please forward it to us. Please send us the exact differentiation command that you typed with all itscommand­line arguments, and the log text that Tapenade produced before it crashes.

If you can send us (a reduced version of) the file that causes the bug, that would be just great!

If you were using the on­line Tapenade, then you got an "Error" web page. At the end is a useful link (for us developpers): "Ifeverything seems correct, it is probably an internal bug. In that case, please send us thisbug report, without modifying the contents of the message. We will examine it promptly.".Please send us this bug report using the button "send us". It will send us the session reference so that we can find its track in ourvery large local log. Most important for that are the session number and the date/time of the session, for example:Session:ac3eQiTVfzwb, Time:Wed Mar 29 07:59:01 MEST 2006

JAVA Version required: I installed the new version of Tapenade and Java now complains.What minimal version of Java jdk is required?

A locally installed version of Tapenade requires a reasonably recent version of Java. See the installation README for details. Forinstance, as of Jan 2008, it requires Java jdk 1.6.

Wrong external's: My differentiated program does not compile because it contains aexternal REALdeclaration.

Tapenade doesn't well recognize the REAL Fortran intrinsic, and therefore considers it as an external. This may also happen for acouple others intrinsics. This bug has still not been fixed. The workaround is simple: just delete the faulty declaration.

Controlling checkpointing in the reverse mode: My reverse­differentiated program is too slow, and I think it comes from checkpointing.How can I tell Tapenade to refrain from checkpointing each and every procedure call ? How can I tell Tapenade to usecheckpointing on a given piece of code ?

By default, Tapenade will apply checkpointing to each and every procedure call, i.e. subroutine or function call. There are variousdirectives that let the user control checkpointing. When used wisely, these techniques can vastly improve the performance of theadjoint differentiated code by modifying the tradeoff between memory use and extra recomputing.

There are three ways to specify places (i.e. procedure calls) where checkpointing must not be done.

You can place a directive in the source just before a particular procedure call. For example:

C$AD NOCHECKPOINT w = toto(x, y, z)

tells Tapenade not to place a checkpoint around this particular call to function toto.You can place a directive in the source just before the definition of a procedure. For example:

C$AD NOCHECKPOINT subroutine titi(a,b)

tells Tapenade not to place a checkpoint around any call to subroutine titi.You can use the command­line argument ­nocheckpoint that takes the list of procedures that must never becheckpointed. For example:

$> tapenade ­b ­nocheckpoint "toto titi" program.f

will checkpoint no call to subroutines toto and titi.

Conversely you can ask Tapenade to apply checkpointing on an arbitrary piece of code.

You can place the pairs of directives described here.Another way, with more effort but that gives you more control, is to make this piece of code a true subroutine: since Tapenadeapplies checkpointing systematically to each subroutine call, you will obtain the desired effect.

Finally there are situations where you can use special predefined checkpointing strategies:

loops with independent iterations,binomial checkpointing, which applies well to long chains of similar iterations such as time­stepping loops.

Independent Iterations Loops ($AD II­LOOP): What is the $AD II­LOOP directive? When can I use it ?

The II­LOOP directive ($AD II­LOOP) tells Tapenade that the following loop has "Independent Iterations", i.e. roughly speaking canbe parallelized. With this information, Tapenade is able to produce a better code in the reverse mode.

A loop can be declared II­LOOP if it has no loop­carried "true" or "flow" data­dependence, i.e. if no loop iteration depends on avalue that is computed by another iteration. The following loop is an II­LOOP:

C$AD II­LOOP DO i=1,N,2 A(i) = 2.0*z A(i+1) = 3.0*z tmp1 = B(T(i)) tmp2 = B(K(i)) vv = SQRT(tmp1*tmp2) C(i) = C(i)+vv ENDDO

because, as you can check, there is no loop­carried true dependence. This implies that the iteration order can be freely shuffled. Onthe other hand, the following loop is not an II­LOOP:

DO i=3,N­2 A(i) = A(i­2) B(i) = B(i+2) prod = prod*C(i) ENDDO

for at least three reasons:

There is a loop­carried flow dependence of distance 2 on AThere is a loop­carried anti dependence of distance 2 on B, and it will become a flow dependence if the iteration order isreversed.There is a loop carried dependence on the product prod

Finally, the following is also an II­LOOP:

C$AD II­LOOP DO i=1,N,2 tmp1 = B(T(i)) tmp2 = B(K(i)) vv = SQRT(tmp1*tmp2) sum = sum+vv ENDDO

Because the loop carried dependence on sum indeed comes from a "sum" reduction, which is OK.

If you add the directive $AD II­LOOP as a comment just before an II­LOOP, you will see that reverse differentiation will produce adifferent code, which uses significantly less PUSH/POP memory space ("tape") than without the directive.More explanations in the paper on Adjoining Independent Computations

User­defined additional checkpoints ($AD CHECKPOINT­START and $AD CHECKPOINT­END): How can I add reverse­mode checkpoints into my source code?How can I force checkpointing of a given piece of code?

Just place Tapenade directives $AD CHECKPOINT­START and $AD CHECKPOINT­END around the piece of code you want tocheckpoint. This allows you to define a checkpoint around a piece of code P. This may prove useful in very long subroutines, thatconsume a lot of memory to store their intermediate variables during the forward sweep.

Adding a user­defined checkpoint around a piec of code P is applicable when P could indeed be extracted as a separateprocedure. In the previous versions of Tapenade, you had to actually make P a procedure to trigger checkpointing on it. Thesedirectives achieve the same effect without forcing you to make this error­prone transformation.

The $AD CHECKPOINT­START must be placed immediately before the first statement that is in P.The $AD CHECKPOINT­END must be placed immediately before the first statement out of P, i.e. the first statement thatfollows the end of P.

Make sure these "first statement in" and "first statement out" are unambiguous. All predecessors of the "first statement in" must beout of P. All predecessors of the "first statement out" must be in P. If necessary, introduce a no­op statement (e.g. CONTINUE) to liftambiguities. There must be no entry into P that does not go through the "first statement in", and no exit from P that does not gothrough the "first statement out". Finally, "first statement in" and "first statement out" must be executable statements, not declarationstatements nor pure control (e.g. closing braces, ELSE, END, ...) User­defined checkpoints can be nested. The following piece ofcode gives an example:

if (x.gt.10.0) then

C$AD CHECKPOINT­START x = x*y if (x.gt.0.0) x =­xC$AD CHECKPOINT­START y = y ­ x*x y = y*yC$AD CHECKPOINT­END x = x + sin(y)C$AD CHECKPOINT­END continue ... endif

It contains two nested checkpoints, both located inside the "true" branch of a conditional. The generated code shows the effect ofthese additional checkpoints. In the forward sweep:

IF (x .GT. 10.0) THEN CALL PUSHREAL4(x) CALL PUSHREAL4(y) x = x*y IF (x .GT. 0.0) x = ­x y = y ­ x*x y = y*y x = x + SIN(y) ... CALL PUSHCONTROL1B(1) ELSE CALL PUSHCONTROL1B(0) END IF

and in the reverse sweep:

CALL POPCONTROL1B(branch) IF (branch .NE. 0) THEN ... CALL POPREAL4(y) CALL POPREAL4(x) CALL PUSHREAL4(x) x = x*y IF (x .GT. 0.0) THEN x = ­x CALL PUSHCONTROL1B(1) ELSE CALL PUSHCONTROL1B(0) END IF CALL PUSHREAL4(y) y = y ­ x*x y = y*y yb = yb + COS(y)*xb CALL LOOKREAL4(y) y = y ­ x*x yb = 2*y*yb xb = xb ­ 2*x*yb CALL POPREAL4(y) CALL POPCONTROL1B(branch) IF (branch .NE. 0) xb = ­xb CALL POPREAL4(x) yb = yb + x*xb xb = y*xb END IF

Binomial Checkpointing ($AD BINOMIAL­CKP): My time­stepping loop fills up the memory space during the forward sweep, and this makes my adjoint code crash.What is the $AD BINOMIAL­CKP directive? When can I use it ?

Binomial checkpointing is the optimal memory/runtime tradeoff for iterative loops, i.e. loops where the computation of iteration ndepends on the results of previous iterations, and therefore there is no hope of making these iterations independent and/or runthem in parallel. A typical example is the time­stepping loop in an instationnary simulation.

Assuming that the final number of iterations N is known, and assuming that each iteration has the same runtime cost, then A.Griewank has given a checkpointing strategy for the iterative loop, which is optimal both in terms of memory used to store thecheckpoints, and in terms of runtime i.e. the number of times a given iteration is repeated. A.Griewank has shown that this so­called"binomial" strategy has a cost in memory and a cost in execution time that grow like log(N). More precisely

The maximum number of checkpoint snapshots (i.e. bunches of values required to restart execution from a given iteration)present in memory at the same time grows like log(N). Therefore the maximum memory required is log(N) times the size ofone snapshot.The maximum number of times a given iteration with be repeatedly executed due to this strategy also grows like log(N).Therefore the time required to compute the adjoint of the N iterations will be (log(N) + Constant) times longer than the time to

run the original N iterations.

In practice, one has less flexibility on the number of snapshots than on the runtime. Therefore one prefers to fix the maximumnumber of snapshots to a fixed q. Then the costs of the binomial scheme change to:

The maximum memory required is q times the size of one snapshot. It is independent of N.The time required to compute the adjoint of the N iterations will grow like the q­th root of N.

This was published by A. Griewank and A. Walther, and first implemented in AdolC. Tapenade is now able to produce an adjointcode that applies this strategy automatically, triggered by the BINOMIAL­CKP directive ($AD BINOMIAL­CKP). Consider thisexample:

C$AD BINOMIAL­CKP nstp­stp1+2 4 stp1 do stp=stp1,nstp x = x+y y = y­x*x v3 = 1.0 do i=3,98 j = i­stp/10 v1 = A(j)*B(i) v2 = B(j)/A(i+1)+v3 A(i) = 2*A(i)+sin(C(j)) B(i­1) = B(i)*B(i+2) ­ A(i+1)*A(i­1) A(i+1) = A(i)+v1*v2 B(i­1) = B(i)+v2*v3 v3 = v3+v2­v1 enddo A(j) = A(i)*B(j) enddo

The outer loop is a good candidate for binomial checkpointing. Its iterations are definitely not independent, so the II­LOOPdirective is out of question. Also, the number of iterations, although dynamic, is known before the loop starts.

The inner loop could also be applied binomial checkpointing, but this would be less profitable as it is a smaller piece of code. Thethree values provided after the BINOMIAL­CKP directive are

The number of iteration steps, or an expression to compute it just before the loop starts. For optimal results, also count theiteration during which loop exit occurs. This is why we wrote nstp­stp1+2 instead of nstp­stp1+1. If the given number ofiterations is wrong, the differentiated code should still work, but will be less efficient in time or memory.The maximum number q of allowed snapshots, chosen by the end­userThe index of the first step in the sequence

Warning: as Tapenade parser is not perfect inside directives, be careful not to put any white space inside these three expressions.These expressions will we copied verbatim in the adjoint code.

If you look at the produced reverse code, you will find that the adjoint code is in fact a driver, initialized with the parameters of theBINOMIAL­CKP directive, and that will trigger the correct sequence of actions to compute the adjoint of the iterative loop. Theseactions are:

Push Snapshot: Push the current memory state on the stack.Look Snapshot: Restore the memory state from the stack, but leave it on the stack.Pop Snapshot: Restore the memory state from the stack and remove it from the stackAdvance: run the next iteration in its original form, without any derivative computed nor value pushed.First Turn: Compute the adjoint of the last iteration and of the rest of the code.Turn: Compute the adjoint of an ordinary iteration (not the last one)

To be executed, the differentiated code must be linked with the definitions of the external routines used by the driver:

TRV_INIT()TRV_NEXT_ACTION()TRV_RESIZE()

They are provided in file treeverse.f (alternatively treeverse.c and treeverse.h) from the ADFirstAidKid. Download First­Aid Kit

Linear Solvers: My original code solves a linear system and the differentiated code for this part of the code is slow.My original code uses an iterative linear solver and the differentiated code produces derivatives whose accuracy gets poorer andpoorer as the differentiated linear solver is used.

Linear solvers are often sophisticated and do not lend themselves to Automatic Differentiation efficiently. Even if you have accessto the source of the linear solver, it may be wiser not to use the AD­generated code for this solver, and write a better one by hand.

Moreover, when the linear solver is iterative, it often happens that the number of iterations required to fully converge the solution yof Ay = b is not enough to converge its derivative yd or yb. Since AD uses the same control as the original program, it stopsiteration when the original program does. Therefore yd or yb may not have reached convergence when y has!

Of course if you use a solver from a library and you don't have access to the source, then you are forced to write the differentiatedsolver by hand. You have no choice. It will just comfort you to learn that this is the best thing to do anyway! Just refer to the FAQitem about Black­box procedures to tell Tapenade about this library call (We would advise you to apply the dummy proceduremethod), and come back here for hints about how to write this hand­coded differentiated linear solver.

Assume that the code to be differentiated has first built a matrix A and a right­hand side vector b, that may be both active. Size of Ais n*n and size of b is n. Assume that some solver is called, e.g.

call SOLVE(A,y,b,n)

that computes the y (of size n) such that:

y is also active. In tangent mode, we can differentiate the above equation by hand and we obtain:

so that we find yd is:

Therefore, since Tapenade probably built a tangent differentiated call looking like:

call SOLVE_D(A,Ad,y,yd,b,bd,n)

you can write your hand­coded SOLVE_D along the lines of:

subroutine SOLVE_D(A,Ad,y,yd,b,bd,n) INTEGER n REAL A(n,n), Ad(n,n) REAL y(n), yd(n), b(n), bd(n) INTEGER i,j REAL RHSd(n)

call SOLVE(A,y,b,n) DO i=1,n RHSd(i) = bd(i) DO j=1,n RHSd(i) = RHSd(i) ­ Ad(i,j)*y(j) ENDDO ENDDO call SOLVE(A,Yd,RHSd,n) END

In reverse mode, it's no surprise things get a little harder! The inputs of the linear solver call are A and b, and the output is y. Bydefinition of the adjoint variables Ab, bb, and yb, we have the dot products equality (dot product noted as (_|_)) for any Ad and bd :

Beware that these dot products are element­wise: the dot product of Ad and Ab is the sum of all Ad(i,j)*Ab(i,j) for all i and j.Replacing yd by its above definition, we get:

for any Ad and bd. Therefore:

The first equation gives directly bb as:

The second equation gives Ab but this is less obvious: we must go down to the indices level:

and we get the value of each element of Ab. Finally, since Tapenade probably built a reverse differentiated call looking like:

call SOLVE_B(A,Ab,y,yb,b,bb,n)

you can write your hand­coded SOLVE_B along the lines of:

subroutine SOLVE_B(A,Ab,y,yb,b,bb,n) INTEGER n REAL A(n,n), Ab(n,n) REAL y(n), yb(n), b(n), bb(n) INTEGER i,j REAL AT(n,n), incrbb(n)

DO i=1,n DO j=1,n AT(i,j) = A(j,i) ENDDO ENDDO call SOLVE(AT, incrbb, yb, n) DO j=1,n bb(j) = bb(j) + incrbb(j) ENDDO call SOLVE(A,y,b,n) DO i=1,n DO j=1,n Ab(i,j) = Ab(i,j) ­ y(j)*incrbb(i) ENDDO ENDDO END

Two remarks:

first, notice that this code increments Ab and bb instead of just setting them. This is the usual adjoint behavior that comesfrom the fact that A and b may very well be used elsewhere in the program and therefore may have other influences on thedependent outputs.Second, we have been considering in these examples that both A and b are active. If either one is not active, then the codesabove may be simplified.

Multiple sessions in parallel: The differentiated code contains parts that do not correspond to anything in my original code.

Maybe you have two web Tapenade sessions active at the same time. When using the Tapenade server on the web, there can beonly one differentiation page opened at a time. Do not try to open two different web browser pages, and call one Tapenade in each,because your Tapenade session is unique: therefore the files you upload would all go into the same directory here, causingstrange behavior.

Approximative source­code correspondence: You claim that the graphical interface shows the correspondence between source and differentiated code. I can't see it!

This source­code correspondence is implemented through html links and anchors, so its behavior is approximate. All it guaranteesis that the target of the link is somewhere visible on the page, but not necessarily in front of the source.

Moreover there are situations where source­code correspondence is lost, for example on declarations. So in this case, the besttarget we can show is the enclosing generated subroutine.

Too many Warning and Error messages: Help! Tapenade sends me plenty of error messages !

Yes Tapenade sends plenty of error messages! One may think there are too many of them. This is because we believe thesemessages can be very helpful in the case when you obtain wrong differentiated values. Sometimes one of these messages is anindication of a real problem for differentiation, so it can save you a lot of time to look at them.

However, maybe you should not try too hard to get rid of all these messages. After all many of them are purely warnings. Very often,Tapenade will produce a correct differentiated code, even if it has complained. We strongly advise you to look at the detaileddiscussion on messages which indicates, for each message, whether it is really important (or not), and why, especially from the ADpoint of view.

Tapenade lets you define in the command line an error level threshold, below which errors are not shown any more.

Finally, to summarize, please do not overlook Tapenade errors because they potentially indicate hard differentiation problems, andat the same time do not be too demanding on the source program: some warnings are there to stay!

Includes and Comments: What a mess! the nice include files of my original program have all been expanded in the differentiated program !What a mess! all my nice comments around declarations have been displaced or lost !

This still happens for some programs. Sorry about that.

For a very long time, you can understand this has not been our topmost priority. After all, it was more urgent to make differentiatedprograms thar run, right?

Also this was not so easy because the internal representation of Tapenade used to digest every declaration into symbol tables thatdid not remember where declarations came from. For the same reason, we had nowhere to keep comments attached todeclarations. This has been improved for version 3. So hopefully, you should notice an improvement in that respect. Sometimeswhen the code uses the implicit feature of Fortran77, the include "calls" can't be placed back. Use implicit none toavoid that.

Despite our efforts, it may happen that some comments are lost. This is a bug and we will be grateful if you signal it to us.

Default sizes of basic data types: My system/compiler has a default size of 8 bytes for REAL, and Tapenade has a default size of 4 bytes.By default, Tapenade assumes that the actual sizes in memory of basic data types are as follows:INTEGER: 4 bytes,REAL: 4 bytes,DOUBLE PRECISION: 8 bytes,COMPLEX: 8 bytes,DOUBLE COMPLEX: 16 bytes,LOGICAL: 4 byte,CHARACTER: 1 byte,

These values are important to solve EQUIVALENCEs, or to match different declarations of the same COMMON block, or moreimportantly for the PUSH/POP mechanism of the reverse mode.

Please check that your system/compiler uses the same sizes. In case of doubt, we provide a small tool to check this. Tapenade letsyou declare new default sizes with the following options:­i2 ­i4 ­i8 ­r4 ­r8 other options might be created as necessary...

Parameterized array sizes: My original file declares array A(N+1), and the code generated by Tapenade declares it as A(201)This is true, and we understand it makes the new code less easy to maintain. When N is a constant, whose value is knownstatically, e.g. 200, Tapenade internally solves the N+1 into its value, here 201. This is on our list of things to improve.

Some cases of "segmentation fault": My differentiated program crashes with a "segmentation fault".

Yes. This may still happen. Sigh...Here are two (not so frequent) cases, that happen in the reverse mode:

There are not as many PUSH'es and POP's because the control flow is not inversed well. The symptom is that the crashoccurs inside a POP, and the stack is empty. If this occurs to you, it is a Tapenade bug. Please signal it to us.If your program passes a FORTRAN PARAMETER (i.e. a constant) as an argument to another procedure, Tapenade tries toPUSH/POP this constant, which results in a segmentation fault at the POP. Until we fix that, don't pass a PARAMETER as anargument.

Multi­directional modes (i.e. option ­multi): What do I do with the code generated with the option ­multi?What is this DIFFSIZES.inc include file used in the differentiated code?What is this extra dimension in the differentiated variables?What is the relation between variables nd, nbdirs and nbdirsmax?The option ­multi turns on multi­directional mode, which produces a code that will compute the derivatives at a single point in theinput space but along several directions in this input space. This definition is for the tangent mode.

Actually, there could be a ­multi behavior in the adjoint mode too. This would compute the gradient for several weightings of theoutputs. At present this multi­directional adjoint mode is not available in Tapenade, but this might be the case in the future.

The way it works is by modifying the structure of the derivative variables. In standard, single­direction mode, a derivative variableretains the structure of its original variable (except for structures, in which non­active components disappear from the differentiatedstructure). In multi­directional mode, derivative variables receive one extra dimension, that corresponds to the multiple possibledirections (or weightings). For instance a scalar variable

real*8 xx

will be differentiated if active into:

real*8 xxd(nbdirsmax)

and an array variable

real*8 aa(100,N)

will be differentiated if active into:

real*8 aad(nbdirsmax,100,N)

Notice that the extra dimension is added on the left, i.e. in the deepest place in the FORTRAN dimensions order. This is useful toisolate the array of derivatives of a given element of aad. Also notice that the (static) size of this new dimension is fixed tonbdirsmax, which must be at least equal to the maximum number of directions you plan to explore at a time.

The differentiated instructions are modified accordingly. A statement such as

a(i1, i2) = 2*b(i1, i2)

will be differentiated as

DO nd=1,nbdirs ad(nd, i1, i2) = 2*bd(nd, i1, i2) ENDDO

Notice that the loop on differentiation directions is placed at the deepest level of the code's loop nesting. Also these loops iteratenbdirs times, thus allowing you to select at run time a number of directions less or equal to nbdirsmax. nd is just the index thatspans on the nbdirs directions. When possible, Tapenade tries to merge these loops on dimensions, to reduce loop overhead.Notice finally that there is no such loop around differentiated procedure calls, as these loops will done inside the differentiatedprocedure.

nbdirs is dynamic, so it is an extra parameter to be passed to the differentiated procedures. on the other hand, nbdirsmax isstatic: you must provide it at compile time. Tapenade strongly suggests that you define it in an include file namedDIFFSIZES.inc. Tapenade adds the include command that includes this file. All you have to do is to create this file. In general,you need to do that only once. For instance, this file may contain

integer nbdirsmax parameter (nbdirsmax = 50)

In Fortran90, a module DIFFSIZES is used instead of an include file. You have to write it so that it defines the constantnbdirsmax.

Getting the Jacobian using the Multi­directional mode: How can I use the Multi­directional mode to compute a Jacobian?

Some easy theory to begin with.Consider a procedure P(X,Y), that evaluates a math function F: X ­> Y. Tangent mode AD of P produces a tangent codeP_D(X,XD,Y,YD) that computes and returns:

the same Y = F(X)and in addition YD = J*XD, where J is the Jacobian of F at point X.

If your goal is to obtain the Jacobian J, and knowing that the tangent code gives you J*XD for any XD, you see that the tangentcode will give you any column of J by just providing a XD which is the corresponding vector of the Cartesian basis of the space ofX's. In other words, if you provide XD full of 0.0 except for a 1.0 at rank i, then you will obtain a YD = J*XD that is the i­th column ofJ. If you do this repeatedly, you will get all the columns of J, and therefore J.

So you need nothing more than the plain tangent mode of AD to obtain J. However, you see that this amounts to running P_Dseveral times for different XD's but for the same X. Some identical computations are going to be duplicated. Multi­directional tangentmode just lets you factor out these identical computations: just ask Tapenade to produce the multi­directional tangent code of P,and then run this multi­directional tangent code with an XD which has now an extra dimension (at the deepest level) so that itcontains all the vectors of the Cartesian basis of the input space (the space of X's).

Technical details of the multi­directional mode are explained here.

We also have in this FAQ a short discussion about Jacobians and Hessians by AD. Please take a look at it, especially in its secondpart devoted to the Jacobian: it contains further discussion about Jacobian sparsity and the multi directional adjoint/reverse mode.

A final warning about the Jacobian built by multi­directional mode: In multi­directional tangent, all derivative variables receivenaturally an extra dimension, which ranges over the different directions of differentiation, i.e. over the dimensions of the input space.For implementation reasons, this extra dimension must be the "deepest" dimension in the new differentiated variable. This isbecause we must be able to easily extract the derivatives of any element of any array. This has a consequence on the shape of thedifferentiated output. Suppose that the output Y is actually an array Y(1:10). The derivative result, which you may think of as theJacobian, will have an extra dimension at the deepest location.

For instance in Fortran, this will be YD(1:nbd,1:10) where nbd is the number of directions, which is (forgetting about sparsity)the dimension of the input space. On the other hand, the second dimension is linked to the dimension of the output space.

You must take this into accout when giving this YD to a procedure that asks for the Jacobian: the conventions on dimensions mustmatch and if they don't you may have to build the transpose of YD or modify the using procedure.

Second Derivatives and Hessians: Can Tapenade return second derivatives?Can Tapenade create a file to calculate Hessian also?

Basically yes! However this requires a bit more interaction with the end­user. Also the resulting differentiated programs could beimproved further if Tapenade had a special mode for second derivatives. There is some ongoing research on these questions inour research team.

The idea to obtain second derivatives is to apply Automatic Differentiation twice. Starting from a procedure P in file p.f thatcomputes y = f(x), a first run of Tapenade e.g. in tangent mode through the command line:

$> tapenade ­d ­head P ­vars "x" ­outvars "y" p.f

returns in file p_d.f a procedure P_D that computes yd = f'(x).xd. Now a new run of Tapenade on the resulting file e.g. in tangentmode again through the command line:

$> tapenade ­d ­head P_D ­vars "x" ­outvars "yd" p_d.f

returns in file p_d_d.f a procedure P_D_D that computes ydd = f''(x).xd.xd0 Specifically if you call P_D_D with inputs xd=1.0 and xd0=1.0 in addition to the current x, you obtain in output ydd the secondderivative f''(x).

TANGENT­ON­TANGENT APPROACH:

In more realistic cases, the input x can be multi­dimensional. The output y can be multi­dimensional too, but this is not so much of aproblem here. If you look for the second partial derivative d2y/dxidxj, you may apply the same two steps, differentiating P withrespect to input xi first, then differentiating the resulting P_D with respect to input xj. If in reality xi or xj are indeed elements of alarger variable, e.g. elements of an array x, you must differentiate with respect to the larger variable. This is because Tapenadedoes not let you differentiate with respect to individual array elements. Then if you call the resulting procedure P_D_D with "correct"inputs xd and xd0, you obtain the second derivative(s) that you want in output ydd. Finally what are the "correct" values toinitialize xd and xd0? In the special case of simple scalars, it is just 1.0. In the general case of array elements, set arrays xd andxd0 to 0.0, except for elements xd(i) and xd0(j) which must be set to 1.0.

Going one step further, one may want several such derivatives d2y/dxidxj for a bunch of xi or xj. The natural answer is then to usetapenade multi­directional mode to compute al these derivatives in just one run. This works fine when multi­directional mode iscalled for only one of the two steps. In other words, what you get then is either a row or a column of the complete Hessian matrix. Inthe general case where one wants to compute the complete Hessian, one needs to call both differentiations in multi­directionalmode. Doing this, you might encouter a couple of simple problems that you will need to fix by hand like we usually do:

The first multi­directional differentiation creates a program that includes a new file DIFFSIZES.inc, containing informationabout array sizes. Precisely, the include file must declare the integer constant nbdirsmax which is the maximum number ofdifferentiation directions that you plan to compute in a single run. nbdirsmax is used in the declarations of the size of thedifferentiated arrays. You must create this file DIFFSIZES.inc before starting the second differentiation step. For instance,this file may contain

integer nbdirsmax parameter (nbdirsmax = 50)

if 50 is the max number of differentiation directions. If what you want is the Hessian, this max number of differentiationdirections is the cumulated sizes of all the inputs x.The second multi­directional differentiation requires a new maximum size value nbdirsmax0, which is a priori different fromnbdirsmax. For the Hessian case, it is probably equal to nbdirsmax. What's more, Tapenade has inlined the 1st levelinclude file, so what you get is a strange looking piece of declarations:

...

INCLUDE 'DIFFSIZES.inc'C Hint: nbdirsmax should be the maximum ... INTEGER nbdirsmax PARAMETER (nbdirsmax=50) ...

We suggest you just remove the INCLUDE 'DIFFSIZES.inc' line, and hand­replace each occurrence of nbdirsmax0by either nbdirsmax or even 50!A complete Hessian is in general symmetric. Tapenade was not able to take advantage of this. You may do so by replacingevery successive lines of the form

... DO nd=1,nbdirs DO nd0=1,nbdirs0 ...

by

... DO nd=1,nbdirs DO nd0=nd,nbdirs0 ...

TANGENT­ON­REVERSE APPROACH:

Suppose the output y is scalar. Then you know that the reverse mode of AD can return you with the Jacobian, which is a single­rowmatrix, i.e. a gradient, at a very low cost. The natural extension of this is to differentiate the reverse­differentiated program, this timein multi­directional tangent mode. Hence the name "Tangent­on­Reverse". In theory, the cost of computing the gradient is a fixedsmall multiple (say 7*) of the cost of the original program P. Similarly, the cost of computing the tangent derivative is also a smallmultiple (say 4*) of the cost of P. If one uses the multi­directional tangent mode, for n simultaneous directions, the cost of the multi­directional tangent derivative is proportional to n (say 4*n). To summarize, the cost of computing the complete Hessian through theTangent­on­Reverse approach should be roughly

4*n*7*P

whereas the cost of computing the complete Hessian through the Tangent­on­Tangent approach should be roughly

4*n*4*n*P

Therefore Tangent­on­Reverse is appealing, although in reality things are not so clear­cut.

Tangent­on­Reverse raises a number of new problems, because of the PUSH and POP that are inserted by reverse differentiation.To obtain an correct tangent differentiation, one must tell the second­step Tapenade about the behavior of these PUSH and POPadded by the first­step Tapenade. This could be improved if Tapenade made the two steps jointly, but this is still not the case...First thing is to declare the inputs and outputs of PUSH and POP. In addition to their argument, they have another hidden parameter,which is the stack. You must declare this in a special file, using the syntax for black­box routines described here. We provide sucha file, named PUSHPOPGeneralLib, in the ADFirstAidKid. You may need to extend it if it doesn't cover a PUSH or POPfunction that your code uses.Download First­Aid Kit

You will tell Tapenade to use this file PUSHPOPGeneralLib in the command line, using the command­line option

... ­ext PUSHPOPGeneralLib ...

The second thing is to link the final Tangent­on­Reverse differentiated code with PUSHPOPDiff.o obtained by compilingPUSHPOPDiff.f.

PUSHPOPGeneralLib declares the inputs and outputs of the PUSH and POP procedures. For example the lines

subroutine popreal8: external: shape:(param 1, common /adstack/[0,*[) type:(modifiedTypeName(modifiers(intCst 8), ident real), arrayType(modifiedTypeName(modifiers(intCst 8), ident real), dimColons(dimColon(none(),none())))) R : (0, 1) W : (1, 1)

mean that POPREAL8 has indeed two arguments. One is the formal parameter, which is a REAL*8, and the other is the stack,whose type is an array of REAL*8 values. The formal parameter is not read but overwritten, the stack is read and overwritteni.e. modified. You may want to add new such declarations if your code uses other PUSH and POP's, such as PUSHREAL4.PUSHPOPGeneralLib also declares the dependencies through the PUSH and POP procedures. For example the lines

concerning POPREAL8

deps: (0, 1, 0, 1)

mean that POPREAL8 outputs a value in its first argument that only depends on the given value of the stack, and also sets anew value for the stack that only depends on the given value of the stack. You may need to add new such declarations if yourcode uses other PUSH and POP's, such as PUSHREAL4.PUSHPOPDiff.f defines the differentiated procedures for the PUSH and POP procedures, in plain tangent mode as well asin multi­directional tangent mode. These differentiated routines exist only for the PUSH and POP of active variables. Inparticular there are no such differentiated routines for PUSHINTEGER nor POPINTEGER. The trick is to push and pop theoriginal variable and the differentiated variable, and since we use a stack, the order of the PUSH'es and POP's is reversed.You may need to add new such definitions if your code uses other PUSH and POP's, such as PUSHREAL4.

This complete process is still very much experimental, so be aware that errors might occur. Don't hesitate to contact us through theTapenade mailing list in case of trouble ( subscribe to the "tapenade­users" mailing list).

Declarations improperly differentiated: The array dimensions are lost in the differentiated declaration.

Common containing arrays: If an array has its bounds declared in the common statement, the differentiated declarations may lost these bounds. If it is the case,you should declare these bounds in the type declaration statement instead of the common statement.

The pointer, target, and allocatable statements:

allocatable :: a(:,:)pointer :: sontarget :: a, y(10)

As described In "Fortran 95/2003 explained", "We believe that it is much clearer to specify these attributes on the type declarationstatements, and therefore do not use these forms." In particular, if these statements use implicit typing, some declarations may be differentiated improperly. Until we fix these bugs, you should completly declare the arrays or allocatable arrays in the typing declaration.

Errors on complicated statements: I got the message (TC43) Could not find type of ..., please checkFor differentiation purposes, Tapenade sometimes needs to split a long and complex expression with subroutine calls, introducinga a number of new intermediate variables. Usually this works fine, but in some situations, Tapenade cannot guess the type andsize of the new intermediate variable. If Tapenade fails to find the correct type REAL of the temporary variable, puts INTEGERinstead, this looses activity. We put the warning message (TC43) to signal the problem. The workaround is to do the splittingbeforehand, on the source program, and to declare the intermediate variables with the correct type.

Copied procedures in the differentiated code : Tapenade generates a large body of what looks like superfluous code that simply duplicates the code going in to Tapenade.

There are nearly­copies of the original procedures in the differentiated code. They appear in general with a new name obtained byappending "_C". This happens in FORTRAN90 when a non­differentiated subroutine USE's a differentiated MODULE. Suppose the original code is:

MODULE M ...END MODULE M

SUBROUTINE SUSE M...END SUBROUTINE S

This code is differentiated as:

MODULE M_B ...END MODULE M_B

SUBROUTINE S_BUSE M_B...END SUBROUTINE S_B

SUBROUTINE S_CBUSE M_B...END SUBROUTINE S_CB

Subroutine "S_CB" is a copy of subroutine "S", with a USE of the differentiated module "M_B" instead of the original module "M".So this does not create a clash for users who use the original code together with the differentiated code: we generate a differentprocedure for "S". So we chose to make a difference between

"S", the original "S" that works with the original "M", and"S_CB", the copy of "S" for the reverse mode, which works with the same module "M_B" as the differentiated "S_B". (the _CBsuffix is built by adding _C in front of the suffix for differentiated variables.)

This causes code almost­duplication but fewer compilation conflicts.However, we are not very satisfied with this strategy and we may modify it some day, from our discussions with end­users...

Building the context for calling the differentiated code : I have differentiated my root procedure. How should I organize the program that calls it?I'm lost with these all these FOO and FOO_D and FOO_CD procedures and modules. How can I include them into my bigapplication?My differentiated program does a segmentation fault because some differentiated globals are not allocated!

The following is yet subject to discussions and modifications, which means it may change sorry !Tapenade differentiates the call tree that is under the root procedure that you have specified. The result is a set of differentiatedmodules and/or differentiated standalone procedures, that do not execute alone but must be used/called from a certain context. It isup to you to write this calling context, and Tapenade can help you only to a certain point. The following discussion focuses on thetangent mode. Be aware that things can be slightly harder in the adjoint mode.

You may use as a basis a calling context designed for the original, non­differentiated root procedure. Notice that even if youprovided Tapenade with the complete calling context code as well, it will differentiate only the call graph below the root procedure.The calling context will not be modified nor adapted to the differentiated root. So even if there is a calling context available for theroot, you will have to modify it by hand so that it can call the differentiated root.

There are many possibile cases, so we'd rather use an example. We'll try to extract general rules later. Here's a small examplesource program. The differentiation root procedure is ROOT. The first column contains the code that is not in the call graph belowROOT.

PROGRAM MAIN USE MMA USE MMB REAL, DIMENSION(:), ALLOCATABLE :: x INTEGER :: i REAL :: cva(2) COMMON /cc1/ cva EXTERNAL GETFLAG CALL GETFLAG(flag) ALLOCATE(x(30)) CALL INITA() DO i=1,20 aa(i) = 100.0/i x(i) = 3.3*i END DO cva(1) = 1.8 cva(2) = 8.1 CALL ROOT(x) PRINT*, 'zz=', zz DEALLOCATE(aa) DEALLOCATE(x)END PROGRAM MAIN

MODULE MMA REAL, DIMENSION(20), ALLOCATABLE :: aa

CONTAINS SUBROUTINE INITA() ALLOCATE(aa) END SUBROUTINE INITAEND MODULE MMA

MODULE MMB INTEGER :: flag REAL :: zzEND MODULE MMB

SUBROUTINE ROOT(x) USE MMA USE MMB REAL, DIMENSION(*) :: x REAL :: v REAL :: cv1, cv2 COMMON /cc1/ cv1, cv2 zz = cv1*cv2 v = SUM(aa) IF (flag .GT. 2) zz = zz*v zz = zz + SUM(x)END SUBROUTINE ROOT

We will differentiate in tangent mode, with head ROOT. We will not specify the name of the output file, so that every module orisolated procedure of the differentiated code will be put into a separate file. The differentiated call graph is returned in filesroot_d.f90, mma_d.f90, and mmb_d.f90.

root_d.f90 defines procedure ROOT_D mma_d.f90, mmb_d.f90 define modules MMA_D, MMB_D

! Diff. of root in tangent mode:! variations of useful results: zz! with respect to varying inputs:! *aa cv1 cv2 x! RW status of diff variables:! zz:out *aa:in cv1:in cv2:in x:inSUBROUTINE ROOT_D(x, xd) USE MMB_D USE MMA_D IMPLICIT NONE REAL, DIMENSION(*) :: x

MODULE MMA_D IMPLICIT NONE REAL, DIMENSION(:), ALLOCATABLE :: aa REAL, DIMENSION(:), ALLOCATABLE :: aad

REAL, DIMENSION(*) :: xd REAL :: v REAL :: vd REAL :: cv1, cv2 REAL :: cv1d, cv2d COMMON /cc1/ cv1, cv2 INTRINSIC SUM COMMON /cc1_d/ cv1d, cv2d zzd = cv1d*cv2 + cv1*cv2d zz = cv1*cv2 vd = SUM(aad) v = SUM(aa) IF (flag .GT. 2) THEN zzd = zzd*v + zz*vd zz = zz*v END IF zzd = zzd + SUM(xd) zz = zz + SUM(x)END SUBROUTINE ROOT_D

CONTAINS SUBROUTINE INITA() IMPLICIT NONE ALLOCATE(aad(20)) ALLOCATE(aa(20)) END SUBROUTINE INITAEND MODULE MMA_D

MODULE MMB_D IMPLICIT NONE INTEGER :: flag REAL :: zz REAL :: zzdEND MODULE MMB_D

The calling context of the original ROOT is not sufficient to call ROOT_D and must be adapted. Here this context consists of programMAIN, but there can be more modules and procedures in it. The new program MAIN_CD produced by Tapenade in filemain_cd.f90 is a possible starting point, but it must be modified by hand. Or you can write it from scratch. For instance:

PROGRAM MAIN_CD USE MMB_D USE MMA_D REAL, DIMENSION(:), ALLOCATABLE :: x REAL, DIMENSION(:), ALLOCATABLE :: xd INTEGER :: i REAL :: cvad(2) COMMON /cc1_d/ cvad REAL :: cva(2) COMMON /cc1/ cva EXTERNAL GETFLAG CALL GETFLAG(flag) ALLOCATE(xd(30)) ALLOCATE(x(30)) CALL INITA() initialize aad,xd,cvad,zzd DO i=1,20 aa(i) = 100.0/i x(i) = 3.3*i END DO cva(1) = 1.8 cva(2) = 8.1 CALL ROOT_D(x, xd) PRINT*, 'zz=', zz, 'and zzd=', zzd DEALLOCATE(aad) DEALLOCATE(aa) DEALLOCATE(xd) DEALLOCATE(x)END PROGRAM MAIN_CD

Places of interest are in blue. Some comments:

the context must call the differentiated root, hence the call to ROOT_D.the context must provide ROOT_D with all the derivatives of its arguments, whether formal parameters or globals. Hence xd,but also aad, /cc1_d/, zzd.These new xd, aad, /cc1_d/, zzd must be declared, allocated if their original variable is, and initialized with thederivatives you want.After return from ROOT_D, the derivatives in these variables can be used, and the variables must be deallocated if theiroriginal variable is.One must make sure that these variables are effectively the same in the context and in ROOT_D, which means that when theyare globals from modules, the same modules must be used on both sides. Hence the USE MMA_D and USE MMB_D. Thisalso ensures that the "good" INIT_A() is called.All these operations need not be done only in MAIN_CD. If MAIN_CD calls utility procedures, you may choose to modify theseprocedures instead. Make sure to keep all these modified procedures separated from their original definition, and link with theoriginal code only when it was not modified. This also applies to MAIN_CD itself.To make sure you don't mix the original context procedures and the new context procedures adapted to ROOT_D, why notgive them the same procedure name? This way, the compiler should complain if you mix them. Here MAIN_CD could justremain MAIN. The same could be said for module names MMA_D and MMB_D but that's another story as currently Tapenadesystematically appends a _D to them.

Tapenade and Windows :

What should I do to use Tapenade on Windows?

Before installing Tapenade, you must check that an up­to­date Java Runtime Environment is installed. If there is none, you candownload and install the latest Java Runtime Environment (Java SE 6 or JDK6) from the java.sun.com site. Tapenade will not runwith an older Java Runtime Environment.

Then

first read our downloading policy, and register either for academic usage or for evaluation usage. You will get a link to aREADME.html file.Looking at the README.html, going to the section about Windows, download the tapenade zip archive into your choseninstallation directory "install_dir"Go to your chosen installation directory "install_dir", and uncompress TAPENADE.Copy "install_dir"\bin\tapenade.bat and update the installation parameters: JAVA_HOME, TAPENADE_HOME, andBROWSER.

Copy the file "tapenade.bat" in a new file "mytapenade.bat" in the subdirectory "bin" located where you have installed the tapenadefiles. In the file "mytapenade.bat" update the variable JAVA_HOME according your java installation, update the variableTAPENADE_HOME: it is where you have installed the tapenade files, update the variable BROWSER, the default is internerexplorer.

Tapenade, Windows and cygwin problem : When parsing a fortran file with tapenade, I get these error messages :

You have multiple copies of cygwin1.dll on your system.Search for cygwin1.dll using ...and delete all but the most recent version. The most recent version *should*reside in x:\cygwin\bin, where 'x' is the drive on which you haveinstalled the cygwin distribution.System: Parsing error

If it is the case, just remove the old cygwin1.dll that is in the directory tapenade/bin/windows.

Tapenade with C programs on Windows, problem with cpp : Preprocessor error

Before parsing a C program, Tapenade calls cpp to preprocess the C program. On a unix machine, cpp is found in the standardPATH. On a windows machine, you can specify the cpp command using the option ­cpp "\...\...\cpp_command". If you have cygwin,you should find the correct path. If you don't find cpp, and if your C program doesn't contain #define, #include, you can call tapenade ­cpp "nocpp" program.c. If yourprogram contains #include they will be ignore.

Validation of the differentiated programs (OLD METHOD) Help, the differentiated program crashes!How can I be sure that the differentiated program really computes what I want?How can I check the derivatives and find the place where the differentiated program goes wrong?

This section is kept here only for the record. It describes the old method that Tapenade provided before 2011. We nowhave a simpler, more powerful method, described here.

In any case, to ease our validation mechanism, we strongly advise you to follow a few preparation steps, before you even start todifferentiate anything. You will understand the reason for these steps if you go through the validation process that follows.

CHECK THE TANGENT DERIVATIVES WITH RESPECT TO DIVIDED DIFFERENCES:

In the following, words between double quotes and drawn in this color (such as "SHAPE") are indeed templates, that must beinstantiated by actual strings depending on the context. Square brackets represent additional arguments, and the star representsrepetition.In the example pieces of programs, actual pieces of code are written in this sort of dark red, short strings in double quotes are"patterns" as above, and

"entiere lines between double quotes and drawn in the pattern color"

represent a piece of code that you must write at that place.

This is the standard way to test the tangent derivatives. Run your original subroutine P with input X, then with X+*Xd, thencompute:

You should get nearly the same result as when running the tangent differentiated subroutine P_d(X,Xd,Y) Differences should beonly on a few last decimals.If this is not the case, here is a way to find where it goes wrong. Use the given library ddTest.f from the AD First­Aid kit.ddTest.f defines subroutines whose name is of the form

DD"TYPE""SHAPE"("(STRING) variable name", "(FORTRAN TYPE) original variable", "(SAME FORTRAN TYPE) differentiated variable" [,"(INTEGER) array length"])

These procedures compare dynamically (at run­time) the derivatives obtained by divided differences (see below how they are built)with the derivatives obtained normally by the tangent differentiated program. If there is a difference, the procedure prints a message.This test is done each time one of these DD* procedures is encountered, on the variable passed as argument.Syntax details:

"TYPE" is one of real4, real8, etc"SHAPE" is ARRAY for an array, and nothing for a scalar.The first argument is any string (no white space please!) in principle, you should put the name of the variable you are testing.This name is only used to be printed!The second argument is the original variable that you want to test.The third argument is its corresponding differentiated variable.For arrays, an extra argument gives the total size of the array, i.e. the product of the sizes of all dimensions.

For example, if at a given point, you want to test the derivative of the REAL*8 x, just insert the call:

CALL DDREAL8("x", x, xd)

To test a whole array, e.g. REAL T(10,0:2), and REAL means REAL*4 on your system

CALL DDREAL4ARRAY("T", t, td, 30)

You may test only one cell of an array, and then you don't need the dimension:

CALL DDREAL4("T_3_2", t(3,2), td(3,2))

You may even cheat a little with array sizes, to check only a fragment of it:

CALL DDREAL4ARRAY("somepartofT", t(3,1), td(3,1), 5)

Notice that these calls must be inserted in the code after the traced variable and its derivative have been computed, and before thetraced variable or its derivative get overwritten later on.

Of course, the above works only on a prepared program. Here is how to prepare it: We assume you have been differentiating asubroutine F. Then the file that uses F_D probably has the shape:

"set all inputs x1, x2, ... to their initial values" "set the differentiation tangent direction in x1d, x2d,..." CALL F_D(x1, x1d, x2, x2d,..., y1, y1d, y2, y2d,...) "use the resulting derivatives y1d, y2d, ..."

Caution: By "inputs" (resp. "outputs") of F, we mean logical inputs (resp. outputs), i.e. all values that will be used inside (resp.returned by) F. This comprises not only the arguments, but any common, global, etc... of any sort that holds a value that is usedinside (resp. returned by) F. Since this verification mechanism needs to perform two runs of F_D on the same inputs, you will needto store all these inputs (we call this "taking a snapshot") before the first run of F_D, and restore them before the second run. If youknow your code very well and you are certain that some inputs are never modified between the 1st and the 2nd run of F_D, thenyou may omit them from the snapshot.

You should prepare the above call site so that it now looks like:

real*8 ddeps, epszero integer ddphase, ddfile common /DDTESTCOMMON/ ddphase, ddfile, ddeps, epszero

"set the inputs x1, x2, ... to their initial values" "take a snapshot "S" of the initial values of x1, x2,..." "set the differentiation tangent direction in variables x1d, x2d,..." "take a snapshot "SD" of this initial direction x1d, x2d,..." ddeps = 1.d­5 "set the inputs x1, x2, ... to a value "plus "" x1 = x1 + ddeps*x1d ... epszero = 0.00000001 ddphase = 1 ddfile = 37 OPEN(37, FILE='epsvalues') CALL F_D(x1, x1d, x2, x2d,..., y1, y1d, y2, y2d,...) CLOSE(37) "reset the inputs x1, x2, ... to their initial values, using "S"" "reset the differentiation tangent direction in x1d, x2d,...using "SD"" ddphase = 2

OPEN(37, FILE='epsvalues') CALL F_D(x1, x1d, x2, x2d,..., y1, y1d, y2, y2d,...) CLOSE(37)

This program will encounter all the DD* calls that are present in F_D. On the first sweep (ddphase=1), it will just store the currentvalue of the variable, which is the value "with " into a temporary file epsvalues. If this file really grows too big, you mightconsider modifying the present strategy, for example building two separate processes that communicate through a UNIX pipe. Onthe second sweep (ddphase=2), the value "with " is retrieved, and combined with the current value to build the "divideddifferences" derivative. This derivative is in turn compared with the AD analytic derivative. if the divided difference is far from the ADderivative (this is parametrized inside ddTest.f, where the percentage of difference is in variable diff), then the word"DIFFERENCE" is printed on the standard output close to the faulty derivatives. Please note that nothing is printed when bothderivatives (DD and AD) are zero. Tapenade has an option that places automatically the calls to all necessary DD* procedures, atthe beginning and end of user­specified procedures. This is of course an option of the tangent mode of AD and its syntax is:

$> tapenade ­d ­traceactivevars "units" ...

The following does it on all units:

$> tapenade ­d ­traceactivevars %all% ...

Of course, you may add extra DD* trace calls by hand after that. Then playing with adding/moving calls to DD* procedures, you will hopefully close in to the place where the F_D file is incorrect!

CHECK THE REVERSE DERIVATIVES USING THE DOT­PRODUCT TEST:

In the following, words between double quotes and drawn in this color (such as "SHAPE") are indeed templates, that must beinstantiated by actual strings depending on the context. Square brackets represent additional arguments, and the star representsrepetition.In the example pieces of programs, actual pieces of code are written in this sort of dark red, short strings in double quotes are"patterns" as above, and

"entiere lines between double quotes and drawn in the pattern color"

represent a piece of code that you must write at that place.

The dot­product test assumes that, if you call J the jacobian of F, and Jt its transpose, F_D(X,Xd) computes actually Yd = J.Xd andF_B(X,Yb) computes actually Xb = Jt.Yb Therefore, for a given Xd, F_D returns Yd = J.Xd, and if we set Yb=Yd F_B(X,Yb) thenreturns Xb = Jt.J.Xd We observe that if the program is correct, then the dot product(Xb | Xd) == (Jt.J.Xd | Xd)should be equal to (J.Xd | J.Xd) == (Yd | Yd)

Tapenade will automatically place the necessary calls inside the differentiated code for you. Use the option ­dptest provided inthe differentiation command. Logically, you should first try to validate the complete differentiated program, whose root is hereprocedure F. Also, the dot­product test requires a preliminary run of a tangent differentiated code, so you should use Tapenade togenerate both a tangent and a reverse differentiated code, both being aware of the dot­product test on F through the ­dptestoption. Finally, the ­dptest option is incompatible with the ­traceactivevars option used to validate the tangent mode, somake sure not to use it now. All in all, you should produce validation codes for F_D and F_B with commands:

$> tapenade ­d ­head F ­dptest "F" ... $> tapenade ­b ­head F ­dptest "F" ...

Look out for (AD14) messages such as: (AD14) Cannot instrument hidden active variable VVV . These tell youthat the preparation of the code for the dot­product test could not be completed, because the test needs to maipulate variable VVVwhich is a global not visible by F. To solve this, you need to make this variable visible inside F, and differentiate again.

Then, to run the dot­product test, you must also prepare your calling program as follows:

INTEGER dpunitcallcount,dptriggercount COMMON /DPCALLCOUNT/dpunitcallcount,dptriggercount

"set the inputs x1, x2, ... to their initial values" "take a snapshot "S" of the initial values of x1, x2,..." "set the differentiation tangent direction in x1d, x2d,..." dptriggercount = 1 CALL F_D(x1, x1d, x2, x2d,..., y1, y1d, y2, y2d,...) "reset the inputs x1, x2, ... to their initial values, using "S"" "assign the y1d, y2d, ... into y1b, y2b,..." CALL F_B(x1, x1b, x2, x2b,..., y1, y1b, y2, y2b,...) CALL DPPRINTSUMMARY()

Be careful to check that the call patterns of the F_D and F_B subroutines match exactly their definition, which you will find in thedifferentiated files. In fact, using the ­dptest option might change the number of arguments of these differentiated subroutines, so

it is important to check that definitions and call match.

Here starts the "Compile­and­Run" step, to which we will cycle later in this explanation...

Compile the test code and link it with the two libraries specific to the dot­product test, dptest.f and dpStack.c. Since this usesthe reverse mode, it should also be linked with the usual adBuffer.f and adStack.c. All these files are provided in the ADFirst­Aid kit.

Running the test code will print on the standard output the values of the dot­products for each fragment between the milestonepoints placed in the code, printing the products (Yd | Yd) when going forward, and the products (Xd | Xb) when goingbackward. There is a summary printed at the end of the process by the call to DPPRINTSUMMARY() in the test site, which shouldmake it easier to read. If there is no bug, corresponding forward and backward dot­products should return the same value, and thevalidation process is complete. If this is not the case, assuming that you have validated the tangent mode as indicated above, thisshows there is a problem with the reverse mode of the fragments for which the dot­products differ.

Then the question is: where does the problem occur? To find this out, we propose that you use a divide and conquer method, bymoving or adding extra milestones for the dot­product test. For example the ­dptest "F" has set only the minimal twomilestones, at the entry and at the exit of F, plus one "somewhere in the middle". These milestones appear in the summary withdefault names: EntryIntof, Inf, and ExitFromf respectively. Choose one (or more...) points in (the control flow of) subroutineF_D. Find the EXACTLY corresponding point in the reverse sweep of subroutine F_B. Then move the piece of code that definesthe Inf milestone of F_D (it is an in­then­endif statement controlled by IF (DPFWDTESTCOUNT('f')) THEN) to the chosennew point in F_D. Alternatively you may duplicate this code (give it a new name e.g 'PointAInf') to place an additionalmilestone. Do the same for F_B. Don't call Tapenade again at this step, because it would overwrite your hand­modified F_D andF_B codes. Compile and run the test again as explained above, from this "Compile­and­Run" step.

If the faulty zone eventually gets reduced to a handful of simple statements, then you should be able to find the bug. Otherwise thefaulty zone may be reduced to a single call to a procedure called, say, G. You must now restart the divide and conquer process thistime on G. Use Tapenade to place the necessary dot­product calls, with the ­dptest option.

$> tapenade ­d ­head F ­dptest "G" ... $> tapenade ­b ­head F ­dptest "G" ...

Look out for (AD14) messages again!Also, the bug may appear not on the first run­time call to G, but on some later call, for instance the 12th. In this case, replace

dptriggercount = 1

in the calling toplevel by

dptriggercount = 12

then resume from the "Compile­and­Run" step.

We advise you not to create new milestones by hand, except by copy­paste as explained above, because this is rather delicate.The order of PUSH and POP's matters, and it is extremely important that no active variable goes through a milestone without beingcaught by one PUSH/POP. This dot­product validation process is much more fragile than the divided differences validation of thetangent derivatives.

Lastly, when all goes well, don't forget to remove the validation options from your differentiation command lines!

All Warning or Error messages : What do all these Warning and Error messages mean?

TAPENADE issues a number of messages resulting from the preliminary analyses done before actual differentiation. Although thetemptation is strong, these messages should not be ignored right away. Especially when AD is concerned, these messages can bethe indication that the program runs into one limitation of the AD technology. So even if the original program compiles and runscorrectly with your compiler, these messages warn you that differntiation may produce a faulty program.

Some messages are requests for help: when TAPENADE needs some help from the user, e.g. because it needs and does notknow the size of an array, it issues a message that asks you to give this size after differentiation is done. Otherwise the resultingprogram will not run.

Here is the list of all Warning and Error messages of Tapenade, grouped by category, and pointing to a discussion on theirmeaning and importance. This list may not be up to date, unfortunately...

Declarations messagesDD01,DD02,DD03,DD04,DD05,DD06, DD07,DD08,DD09,DD10,DD11,DD12, DD13Type messagesTC01,TC02,TC03,TC04, TC05,TC06,TC07,TC08,TC09, TC10,TC11,TC12,TC13,TC14, TC15,TC16,TC17,TC18,TC19,TC20,TC21,TC22,TC23,TC24, TC25,TC26,TC27,TC28,TC29, TC30,TC31,TC32,TC33,TC34, TC35,TC36,TC37,TC38,TC40,TC41,TC42, TC50,TC51,TC52Control flow messagesCF01,CF02,CF03,CF04,CF05Data flow messages

DF01,DF02,DF03,DF04Automatic Differentiation messagesAD01,AD02,AD03,AD04,AD05,AD06, AD07,AD08,AD09, AD10

Declarations problems:(DD01) Cannot combine successive declarations of variable var: Type1 and Type2 (ignored new)(DD02) Cannot combine successive declarations of character array: Type1 and Type2 (ignored new)(DD03) Cannot combine successive declarations of constant const type: Type1 and Type2 (ignored new)(DD04) Double definition of function Func (ignored new)(DD05) Cannot combine successive declarations of function Func (ignored new)(DD06) Cannot combine successive declarations of function Func return type: Type1 and Type2 (overwritten previous)(DD07) Cannot combine successive declarations of function Func kind: kind1 and kind2 (ignored new)(DD08) Double declaration of external function Func(DD09) Double declaration of intrinsic function Func(DD10) Double declaration of the main program: Func1 and Func2 (overwritten previous)(DD11) Double declaration of function Func in library file LibraryFile(DD12) Cannot combine successive declarations of type Type (ignored new)(DD13) Double definition of label label (ignored new)

There are two successive definitions or partial declarations of some symbol, as a variable, constant, type, label or functionname. These successive declarations occur in the same scope and do not combine into a coherent declaration for thesymbol. TAPENADE therefore chooses either to ignore the new piece of declaration, or to overwrite the previous piece ofdeclaration, as indicated by the message. For DD08 and DD09, the two choices are equivalent.

Why should you care? TAPENADE chooses to ignore the indicated declaration or part of your original program. Thereforethe types considered by TAPENADE may not be the ones you expect, or the program actually differentiated may differ fromthe program you think. In particular when types are concerned, remember that differentiation occurs only on REAL (orCOMPLEX) typed variables. Therefore some variable may have no derivative whereas you think it should have one. Also forDD13 or DD10, the flow of control may be different from what you think, or from what your local compiler used to build. Alsofor DD11, some information that you give about an external function may be ignored and replaced by another one.

What can you do? Remove the declaration or definition in excess, or modify conflicting declarations so that they combinewell into the type that you expect. For DD11, clean up the library file by merging the data about a given function into onesingle entry.

Type problems:(TC01) Expression is expected to be numeric, and is here Type(TC02) Expression is expected to be boolean, and is here Type(TC03) Expression is expected to be a record, and is here Type(TC04) Expression is expected to be a pointer, and is here Type(TC05) Expression var is expected to be an array, and is here Type(TC06) Loop index is expected to be integer, and is here Type(TC07) Loop bound is expected to be integer, and is here Type(TC08) Loop times expression is expected to be integer, and is here Type(TC09) Real part of complex constructor is expected to be numeric, and is here Type(TC10) Imaginary part of complex constructor is expected to be numeric, and is here Type(TC11) Argument of arithmetic operator is expected to be numeric, and is here Type(TC12) Array index is expected to be numeric, and is here Type(TC13) Triplet element is expected to be numeric, and is here Type(TC14) Substring index is expected to be numeric, and is here Type(TC15) Argument of logical operator is expected to be boolean, and is here Type

The indicated expression is used in a context which requires some type. Using the available declarations, the type­checkerfound that this expression actually has a different type, indicated in the message.

Why should you care? This may be the sign of an error in the program. Even if this original program actually compiles andruns well, remember that differentiation occurs only on REAL (or COMPLEX) typed variables. If some sub­expressionbecomes of another type, there will be no derivative associated, even if you think there should be one. Remember also thatthese problems make the program non­portable.

What can you do? Check the types. You may also use the appropriate conversion functions. Be aware that when youconvert a REAL into an INTEGER, the differentials are reset to zero.

(TC16) Type mismatch in assignment: Type1 receives Type2(TC17) Type mismatch in case: Type1 compared to Type2(TC18) Type mismatch in comparison: Type1 compared to Type2

The two terms in the assignment or comparison, or the switch and case expressions, must have compatible types. Otherwise,the meaning of the instruction is not well defined, and not portable.

Why should you care? This can be the indication for a standard type error and, like above, this can lose derivatives.

What can you do? Check the types. Use conversion functions when appropriate.

(TC19) Equality test on reals is not reliable

Equality and non­equality are not well defined between REAL's, because they are defined only up to a given "machineprecision". Equality test can yield different results on different machines.

Why should you care? As far as AD is concerned, equality tests on REAL's are sometimes used as the stopping criterion forsome iterative process. Then it relates to a well­known problem: do the derivatives converge when the function does, and ifyes, do they converge at the same speed?

What can you do? Check that this does not impact differentiation. Take care to perform the usual validation tests on thecomputed derivatives.

(TC20) Variable var is not declared(TC21) No implicit rule available to complete the declaration of variable var(TC22) Variable var, declared implicitly as Type1, is used as Type2(TC23) Symbol var, formerly used as a variable, now used for another object(TC24) Wrong number of dimensions for array Array: n2 expected, and n1 given(TC25) Dimensions mismatch in vector expression, Type1 combined with Type2(TC26) Incorrect equivalence(TC27) End of common /COMMON/ reached before variable var(TC28) Variable var cannot be added to common /COMMON1/ because it is already in common /COMMON2/(TC29) Common /COMMON1/ declared with two different sizes: s1 vs s2(TC30) Type mismatch in argument Arg of function Func, expects Type1, receives Type2(TC31) Type mismatch in argument Arg of function Func, declared Type1, previously given Type2(TC32) Wrong number of arguments calling Func: n2 expected, and n1 given

These messages indicate conflicts between declaration and usage of some variable. Message TC25 occurs in the context ofvectorized programs. Messages TC26 to TC29 indicate a problem in EQUIVALENCE's and COMMON's. Recall thatvariables in a COMMON are stored contiguously, so that their relative order cannot be changed, and extra variables cannotbe inserted occasionaly. Also a variable cannot be in two different COMMON's, and one cannot set an EQUIVALENCEbetween two variables that already have their own, different, memory space allocated. Messages TC30 to TC32 indicate amismatch between the formal and actual parameters of a function. Recall that the standards require that formal and actualparameters match in number and size.

Why should you care? In general, if you let the system choose the type of a variable, you run the risk that it becomes REALwhile you don't want, or the other way round. And this in turn impacts differentiation. Messages TC24, and TC26 to TC32 areeven more serious: They can indicate that the code relies on FORTRAN weak typing to perform hidden equivalence'sbetween two arrays, or to resize, reshape, or split an array into pieces. Since TAPENADE cannot possibly understand what isintended, it might not propagate differentiability and derivatives from the old array shape to the new array shape.

What can you do? Declare variables explicitly. Try to avoid the "implicit" declaration facility. Declare arrays with their actualsize (not "1"). Never resize, reshape, or split arrays implicitly through parameter passing, and avoid to do so acrossCOMMON's

(TC33) Intrinsic function Func is not declared(TC34) External function Func is not declared

These messages indicate that you did not provide TAPENADE with the sources of the indicated procedures.

Why should you care? If TAPENADE doesn't see the source of a called procedure, it is obliged to make several worst­caseassumptions about it. Moreover, it this procedure must be differentiated, TAPENADE can't do it. You will have to do ityourself. In some cases this can be profitable because an expert user can do a better job than TAPENADE.

What can you do? If you just forgot the procedure's source file, add it to the list of source files. If the procedure is missing fora good reason, first give TAPENADE type and data­flow information about the missing procedures, using either a dummysource procedure, or using our library mechanism. This can only make things better, as the conservative worst­caseassumptions often have a negative impact. Then think about these missing procedures: if it turns out they will have aderivative, you will have to provide this derivative: message AD09 will remind you of that later. In the best situation, you areable to write an analytic differentiated code yourself. In the worst situation, you may use divided differences for the derivativeof this unknown procedure, but this will loose some accuracy!

(TC35) Subroutine Func used as a function(TC36) Type function Func used as a subroutine

(TC37) Return type of Func set by implicit rule to Type

These messages indicate conflicts between declaration and usage of some function. Programs with messages TC35 andTC36 should not compile.

Why should you care? In general, if you let the system choose the return type of a function, you run the risk that it becomesREAL while you don't want, or the other way round. And this in turn impacts differentiation.

What can you do? Insert the correct declaration.

(TC38) Recursivity: Func1­­>Func2­­>...­­>FuncN

Recursivity is when a subroutine Func1 calls a subroutine Func2, which itself calls a subroutine Func3, and so on, and whenthis eventually leads to calling Func1 again. In other words this is a cycle in the call graph. Basically, this is not an error,although FORTRAN77 used to forbid recursive programs.

Why should you care? It is just a matter of time. Recursive programs are harder to analyze. We are progressively updatingTAPENADE analyses so that they cope with recursivity.

What can you do? In the meantime, check that the resulting derivatives are correct, using the classical tests.

(TC40) No field named name in Type(TC41) Recursive type definition

These are illegal uses of types. TC40 says the program looks for a field name which does not exist, and TC41 that sometypes directly contains a field of the same type, thus yielding an infinite size!

Why should you care? It is surprising is the original program actually compiles! Differentiation is undefined.

What can you do? Rewrite the definitions of these types.

(TC42) Definition of type Type comes a little too late

The definition of the type came after it was used. This is just a warning, because TAPENADE will do as if the definition wasdone in due time. But you should better fix this.

(TC50) Label label is not defined(TC51) Label variable var must be written by an ASSIGN­TO construct(TC52) Variable var used in assigned­GOTO is not a label variable

These are illegal uses of labels, or label variables.

Why should you care? If labels are misunderstood, the final flow of control that TAPENADE understands and regeneratesmight not be the one you expect. Therefore TAPENADE does not differentiate the program you expect.

What can you do? Define the labels you use. Use the correct constructs to manipulate label variables.

Control flow problems:(CF01) Irreducible entry into loop(CF02) Computed­GOTO without legal destinations list(CF03) This assigned­GOTO to var goes nowhere(CF04) This assigned­GOTO to var only goes to label label(CF05) This assigned­GOTO to var is possibly undefined

These messages indicate problems in the flow of control of the program. Message CF01 comes from jumps that go fromoutside to inside a loop, without going through the loop control header. The meaning of this is not well defined and isprobably implementation dependent.

Why should you care? Of course a bad control of a program yields a bad control of its differentiated programs.

What can you do? Avoid irreducible loops. Avoid jumps into loops and into branches of a conditional structure. Try to usestructured programming instead. If an assigned­GOTO goes to only one place, at least replace it by a GOTO.

Data flow problems:(DF01) Illegal non­reference value for output argument Arg of function Func(DF02) Potential aliasing in calling function Func, between arguments arguments(DF03) Variable var is used before initialized(DF04) Variable var may be used before initialized

These problems are detected through the built­in interprocedural data­flow analysis of TAPENADE.

Why should you care? Programs with DF01 usually provoke segmentation faults. Aliasing (DF02) is a major source oferrors for DIfferentiation in the reverse mode, as described in more detail in the "Aliasing" section. Uninitialized variables leadto uninitialized derivatives.

What can you do? Output formal arguments of a subroutine must be passed reference (i.e. writable) actual arguments. It isbetter to avoid aliasing. When aliasing is really a problem, one can easily avoid it by introducing temporary variables, so thatmemory locations of the parameters do not overlap. Variables should be initialized before they are used.

Automatic Differentiation problems:(AD01) Actual argument Arg of Func is active while formal argument is non­differentiable(AD02) Actual output Arg of Func is useful while formal result is non­differentiable

These two messages are usually associated. They indicate the following situation: Some programs implicitly convert REAL'sto INTEGER's during a function call, then these INTEGERS follow their way through the code, and finally are converted backinto REAL's during another function call.

Why should you care? If the original REAL's are active, i.e. depend on the independent input variables, and the finalREAL's are useful, i.e. influence the dependent output variables, then this temprary conversion into INTEGER's has lostdifferentiability and derivatives. Differentiability is lost, derivatives are wrong!

What can you do? REAL's must remain REAL's. If absolutely necessary, there is a "secret" option in TAPENADE to actuallydifferentiate these strange REAL­INTEGER's...

(AD03) Active variable var written by I­O to file file(AD04) Useful variable var read by I­O from file file

This is comparable to the above AD01 and AD02. These two messages are usually associated or, to put it differently, if thetwo messages occur for the same differentiation session, then there is a risk. The risk occurs in the following situation: Someprograms write intermediate REAL values into a file and later on, read back these REAL values from this file. This processlooses differentiability.

Why should you care? If the written value is active, i.e. depends on the independent input variables, and the corresponding(equal) value read is useful, i.e. influences the dependent output variables, then this temporary storage into a file has lostdifferentiability and derivatives. Differentiability is lost, derivatives are wrong! For instance if the code looks like:

subroutine TOP(x,y,fileno) real x,y,tmp integer fileno write(fileno,*) x ... read(fileno,*) tmp y = 2.0*x + tmp end

Tapenade cannot find the dependency path that goes from the input x through de file "fileno" and finally to the output y. if xd is1.0 yd will be 2.0, whereas the correct answer should be 3.0.

What can you do? Do not use files as temporary storage. Use arrays instead. You may also use TAPENADE "Black­box"mechanism to disconnect differentiation on the calling subroutines, and differentiate them by hand, with an ad­hocmechanism to propagate derivatives.

(AD05) Active variable var overwritten by I­O

This reminds you that a variable that was active has been overwritten by an I­O operation

Why should you care? The derivative is of course reset to zero. Maybe this is not what you expected?

What can you do? Check that it is OK. Otherwise try to put all I­O operations outside the part of the program you are actuallydifferentiating.

(AD06) Differentiate of function Func needs to save undeclared side­effect variables: variables(AD07) This call needs to save undeclared side­effect variable: extern­var(AD10) call to Proc2 in differentiation of procedure Proc1 need to save and restore the I­O state

These messages all indicate that the checkpointing mechanism is not certain that it can do a correct job. These messagespoint at (a collection) of procedure calls in the program, e.g. all calls to Func, or one particular call, or procedure (i.e.subroutine or function) Proc1 calling another procedure (i.e. subroutine or function) Proc2.

In reverse mode, the standard strategy of Tapenade is to checkpoint all procedure calls (cf a quick description ofcheckpointing, or Tapenade User's guide for something more technical). Checkpointing a call to Proc2 inside Proc1 meansthat the reverse differentiated subroutine of Proc1 is going to call Proc2 twice, or more precisely the original Proc2 once andthen the differentiated Proc2. Roughly speking, these two calls must be made with the same inputs, so that the 2nd call is anexact duplication of the 1st call. To do this, a minimal "environment" needs to be stored just before the 1st call to Proc2 and

restored just before the 2nd call. We know how to save and restore most variables, e.g. the formal parameters of Proc2, butalso the globals known by Proc1 and Proc2. On the other hand, we don't know how to save and restore hidden SAVEvariables, or private global variables of a module, or a COMMON which is not declared at the calling subroutine level, or theposition of the read/write pointer inside an opened file which is being read/written, or the very open/close status of a file, andseveral other things... These messages just signal such (possibly) nasty situations.

Why should you care? Checkpointing cannot restore the environment before the second call to Proc2. Therefore thedifferentiated program might be wrong.

What can you do? Either make absolutely sure that this is not a problem in the present case, because you know that thesecond call to Proc2 will behave just like the 1st one, at least as far as differentiation is concerned. You need to know thecode for that because errors will be hard to track down. For instance, one favorable situation is when the (AD10) messagecorresponds only to prints of messages on the screen. The messages will appear twice, but derivatives should be correct.Or modify your code so that the problem disappears, for instance insert the COMMON declaration at the level of the callingsubroutine. or transform the SAVE variables into elements of a COMMON that everyone can see, or take the file I/O ordynamic memory allocation or deallocation outside the code to differentiate,Or (more high­tech!) tell Tapenade not to checkpoint the calls that raise the problem using the "nocheckpoint" directive.

(AD08) Don't know the size of dimension n of array array, which must be saved

In some cases, for example in reverse­mode AD, the differentiated function must save some variables, to restore them later. Ifone such variable is an array, and the size of this array is dynamic, TAPENADE must know this size to actually do the save.

Why should you care? If you don't give this size, the differentiated program will not compile.

What can you do? Define the value of the PARAMETERS that TAPENADE requests, in a special include file. You may alsoexplicitly give this size in the original file, so that the message does not show up!

(AD09) Please provide a differential of function Func for arguments arguments

You must provide the differentiated program with a new function, differentiated from Func with respect to the input and outputparameters of Func specified in the arguments. Probably Func is an external function, for which TAPENADE has no source,and therefore it couldn't differentiate it.

Why should you care? If you don't give this new function, the differentiated program will probably not compile.

What can you do? Define this new function. You have the choice of the method. Maybe you know explicitly the derivativemathematical function, which can be implemented efficiently. Maybe you have the source and you may use AD again, withTAPENADE or with another tool. Maybe you just want to run a local "divided differences" method, knowing that this mightslightly degrade the precision of the derivatives. In any case, make sure that you provide a differentiated function in the correctmode (tangent, reverse...), and which adheres to the conventions of TAPENADE about the name of the function and theorder of its parameters.

... To be continued ...