app: move all special-case mode processing optimizations to GimpOperationLayerMode

Stuff like passing "input" directly if "aux"'s opacity is 0, etc.
Used to be partly handled by normal mode, even though it applies
to other modes too.

Adjust the logic for the new compositing modes.

Add a GimpLayerModeAffectMask enum, and a corresponding
get_affect_mask() function to GimpOperationLayerMode, which
specifies which of the op's inputs, if any, are affected by the
mode, apart from the overlapping regions.  Most modes affect only
the overlapping regions, but dissolve and replace also affect the
rest of the input.  This information is used for determining if
the optimizations are applicable.

app/operations/operations-enums.h

@@ -47,4 +47,17 @@ typedef enum
 } GimpLayerCompositeMode;
 
 
+/*
+ * non-registered enums; register them if needed
+ */
+
+
+typedef enum  /*< pdb-skip, skip >*/
+{
+  GIMP_LAYER_MODE_AFFECT_NONE = 0,
+  GIMP_LAYER_MODE_AFFECT_DST  = 1 << 0,
+  GIMP_LAYER_MODE_AFFECT_SRC  = 1 << 1
+} GimpLayerModeAffectMask;
+
+
 #endif /* __OPERATIONS_ENUMS_H__ */