You can open a closed mask in After Effects by right-clicking on the mask path and selecting "Mask and Shape Path" > "Closed". This will toggle the mask from closed to open.
Another way is to use the Direct Selection Tool (A), click on the segment between the two anchor points you want to open, and then hit delete. This will break the path at that point, making it open.
For masks that are drawn open and then closed, you can use the Pen Tool to select the two end points of the path and then hold down Control (Command on Mac) while clicking on one of the end points, followed by clicking on the other end point. This will close the mask.
Remember: An open path cannot create transparent areas for a layer, but is useful as a parameter for an effect.
