A Custom Control is a WPF Control class that is derived from UIElement or one of its derivations. If you want to implement a special Button Control you could decide to derive from Button. If you want to implement a Control that manages a collection of objects (e.g. a list control) you can derive from ItemsControl.
It depends on your requirements what base class to choose.
In contrast to a UserControl a Custom Control has a ControlTemplate which can be modified by a developer that uses your Control.
See Custom Controls